-: Okay, and we are now recording.
Good morning, my name is Andrew Cress
and I am the president of the California Association
for Institutional Research Board,
affectionately known as CAIR.
Thanks so much for joining us today
for our fourth installment of the CAIR data talks series.
These online workshops and seminars are an opportunity
for the community to connect with new tools and practices
that enhance the IR profession's mission
of supporting evidence based decision making.
The recording of the session will be made available
on the CAIR webpage at a link that I'll provide to you
in the chat.
Today with us, we have several colleagues
from Plaid Analytics,
which is a partner organization of CAIR's.
I'll let them introduce themselves in just a minute.
They're going to be demonstrating for us today
an open source tool called DataHub
that they're going to talk to you much more in depth about.
It supports data governance, data integrity,
things that we all care about I know in the CAIR community.
In the latter half of the session,
they're going to be guiding us through a hands-on
interactive tutorial on DataHub.
So start stretching your typing fingers
and be ready for interacting with us during that experience.
If sessions like this sort of interest to you,
I encourage you to sign up for the CAIR Listserv
which you can do on our website, CAIR.org.
And you can receive announcements
about additional professional development opportunities
like this in addition to our upcoming conference
in LA Jolla, California.
I don't know if you can see behind me,
but there's people that appear
to be having lots of fun parasailing.
And so you can come visit us in November 16th
through 18th 2022 and parasail,
and also maybe get some great
professional development experience at the same time.
If you're interested in that,
please go ahead and navigate over
to CAIR.org/CAIR2022, CAIR ad analytics.
-: All right, thank you Andrew for the kind introduction.
I'll introduce myself as the other Andrew,
but you can call me Drinkwater
'cause that's easy to say and pretty unique.
I'm Andrew Drinkwater,
I'm the president of Plaid Analytics.
We are an analytics firm based in Vancouver, Canada,
and we help institutions across North America
advance their analytics.
For us, this takes a few different forms,
primarily around data engineering, visualization,
enrollment, forecasting, and then the strategy
and governance to make those things successful.
All right, so today we're going to talk
about supporting data governance
with a tool called DataHub.
And we came across DataHub earlier this year
as a happy coincidence from a project
that we were working on in California
with community college,
looking to help them better be able to document
the information that they were viewing
through Tableau dashboards, ultimately understanding
where does that information come from.
So in the case of that project,
it involves the Tableau dashboards as mentioned,
there are data sources on Tableau server,
some of them are shared, some of them are embedded.
Those are sourced from a Data Warehouse
that we helped them build.
Those themselves come from multiple different systems
but primarily the Banner Student Information System,
some spreadsheets and occasionally the odd cloud system
can be worked into the mix as well.
And DataHub was a way of being able to manage
all that different information from a metadata perspective.
So what information is included in say the Data Warehouse
or where does it come from in Banner,
what's used in Tableau, et cetera,
and to be able to provide some visibility into that.
So that project is in progress.
It's not yet complete,
but we're working with DataHub as a way of providing
all this documentation.
Other Andrew, this is your reminder to start the recording.
Andrew Cress: Thank you, we're good. We're good.
-: Okay, so what are we going to talk about today?
We'll introduce who Plaid is
and who the team is that's on the line today.
We'll introduce what DataHub is in a little bit more detail
than what I talked about here.
And then we will do a hands-on workshop
for those interested, which will take about 45 minutes.
Questions are welcome anytime,
but we do have a Q&A block towards the end.
So what does Plaid do?
As an example, we talked about earlier,
we worked with community colleges and universities
to build automated data pipelines, Data Warehouses,
and data lakes that are higher ed focused
and enrollment forecasted tools.
To be successful with those things,
we need strategy and governance wrapped around them.
And so the governance component
is what we're talking about today with DataHub.
We work with leading institutions across North America
to build their analytics capacity.
Here's a few examples.
I'm Andrew Drinkwater,
I'm the president and co-founder of Plaid Analytics.
And prior to launching Plaid
about five and a half years ago,
I worked for the University of British Columbia in Canada.
I was a senior planning and research analyst there
responsible for tuition and enrollment forecasting
and admissions data warehousing.
I got my start in higher ed working as an academic advisor
in the early 2000s, helping students navigate,
shutting down one university,
absorbing it into another and changing the programs
which was a fascinating way to start my career
and ultimately led into my data experience with Tableau
as the start of my journey
towards where we find ourselves here today.
I will turn it over to my colleague Pat
to introduce himself and Clara.
-: Good morning, I'm Pat Lougheed.
I'm the other co-founder at Plaid.
I have spent almost 25 years now
in various different higher education institutions
in different areas,
from Information Technology Services for academic units
to Central Student Services, to Institutional Research,
and lots of other places in between.
-: I'm Clara, I'm analytics architect here at Plaid.
So most of my background is with higher education
and ERP implementation.
And right now I'm working with these projects with Plaid.
-: Thanks, Clara.
Clara's also very modest.
She's a national and international champion at cheerleading,
recently returned from the World Championship in Florida,
taking home 5th place in the world,
and put Canada 3rd on the map.
So congratulations to Clara,
and sorry for embarrassing you in this live talk.
Patrick: First on the map in Canada.
-: First, yes.
-: First on the map in Canada. -: First in Canada.
-: Right, yes.
Canada 3rd place in the world is what I meant to say there.
Congratulations to you and your team.
It's an incredible achievement.
We're here today to talk about a tool called DataHub.
And this is an open source metadata management tool.
It's effectively a spin-off,
if you can call an open source tool that from LinkedIn.
And so LinkedIn had these challenges
with how to manage myriad data from different databases
and cloud systems and information about users and staff
and how this was used in dashboards, et cetera, et cetera.
And so the need is not actually that different
than what we talked about higher education,
it's just different data.
And so they have over the last, I believe it's year or so,
open source this tool.
And so it's still supported by LinkedIn
through various developers.
There's also a couple of other companies
who are very active in supporting this tool.
Primarily Acrodata is the one leading
the open source initiative right now.
There's a few others, and of course Plaid is helping
where we count as well in particular
with the new integration with Tableau
that we will show today.
Pat was instrumental in finding some of the bugs
in the earliest releases of that
so that it would work as well as hopefully
it will live today.
So thanks to Pat for contributing back
to the community as well.
So DataHub supports a number of different things
that you'll see today.
It supports search and discovery of datasets.
So we'll walk through some examples of how to find datasets
that are from say the Banner Student Information System
or the Data Warehouse or Tableau
and see how those relate to other pieces in the system.
It'll also help you understand Lineage,
which is my favorite feature.
So being able to work my way backwards
from what's in this visualization
to where did that data come from in Tableau
to where did that data come from in the Data Warehouse
to where if it's applicable, did that come from in Banner.
To me, this is really, really cool.
So I'm a big fan of that part.
It also sports documentation and tags
which is very necessary.
So what does this calculated field mean?
Or what does this field from the source system mean?
Maybe there's some fields that probably shouldn't be used
and maybe we want to have that in the documentation as well.
It supports business glossaries built upon that
and then it also has support for data governance roles
and user access.
So if you think a committee or organization
within your university or college
that's responsible for data governance,
those roles are supported in different ways
and you can also manage the visibility
of what different users have access to.
And then there's lots of other features.
The one other thing I'll mention
is that they're innovating super, super fast.
There's a new release just about every week
with some interesting new features every time,
so this list will change very quickly would be my guess.
This is just an example.
So I mentioned Lineage is my favorite feature.
This helps us understand where a turntable from Banner
gets used in the Data Warehouse
is kind of the first column there and then...
Or sorry, in the data integration tool is the first column,
second column is the Data Warehouse, roughly speaking,
third stage is some further integrations
and then fourth stage is the actual end result tables
that we might use to support our visualizations
which come further downstream.
Okay so in our view,
there's some major pluses to a tool like DataHub.
It is open source by nature,
so if you want to get started on your own with open source
and you're familiar with that community,
and especially if you've got some programming chops,
it is awesome.
There's also options to have it hosted for you.
We'll talk about some of those later today as well.
So if you don't want to manage all of it,
there are other options available.
I mentioned innovation, they are moving so fast.
So three months ago they didn't support Tableau.
We were trying to figure out how to do it,
and now they have really robust information about Tableau
as an example.
You can connect to a wide variety of different systems
whether they're relational database systems
or visualization tools or cloud tools, et cetera,
there's lots.
This helps you promote trust in your data
by making it more visible to your users
what different information is, where it comes from.
And this can support your data governance initiatives.
Okay so we are going to do a really quick demonstration
of what the tool is for those who can only join us
for half an hour.
And then after that, I'll turn it over to Pat
for the more detailed interactive workshops.
So no need to follow along at this point unless you want to
but this will give a brief overview of what the tool is.
When you have created your account with Strigo
and you log in,
you should be presented that looks something like this.
I'll note here that the "Lab is Reconnecting" piece
is very common and basically don't need to worry
about it for now.
So what we're going to be doing is we're taking a small bit
of information from here
and we're going to open a new browser tab.
You may also see an error message
like "Something went wrong".
This is our first time using this platform
and we found the work around,
but we still need to figure out the "What went wrong" piece.
So our apologies for that part,
but what we're going to do when we log in
is we're going to click the gear icon
and then the machine info,
and we're going to copy the dynamic DNS
and Pat will go through this in more detail with you later.
And then we're going to open a new browser tab
and then we'll be presented with the DataHub screen.
Once you've got this for a login,
you'll log in using the username and password provided,
looks like I spelled wrong.
So it should be DataHub and DataHub, apologies for that.
Once you log in, you'll be greeted with a screen
that looks like this.
It knows that you're new.
So here I should stress that my environment for this
is different from your environment for this
is different for Pat's, is different from Andrew's,
et cetera, but they all have the same foundation.
So you can do whatever you want here,
you won't break it for anybody else.
Go for it.
So what we're going to do from here is we're going to start out.
What we can see is that the "Explore your Metadata"
has no metadata found.
So right now this is an empty shell,
there's no information in it.
What we're going to do as a starting point here
is we are going to ingest some information.
And so if we click on Ingestion
which was up at the top right hand side there,
we'll see that we have a number of different data sources
that we've created for you in advance.
So we've created a shell of a Banner database
and a shell of a Data Warehouse
using the MySQL Relational Database Management System.
You'll note that this is MySQL
rather than something like Oracle,
and that's because MySQL is already built into DataHub,
we don't have to install any additional packages
but it does support Oracle and SQL server and so on.
So what we're going to do is we're going to ingest Banner first.
So this is Banner using air quotes.
This is a demonstration database that Plaid has
that has a variety of tables
that are mostly looking like Banner
and have a bunch of fake data in them.
However for DataHub's purposes,
we're not ingesting any actual data,
we're just ingesting the structural information,
so what tables, what columns,
what types are they, et cetera.
So we'll start by hitting Execute.
And as a reminder, we'll go through this in much more detail
following this.
It has been submitted and it's now running.
This usually goes pretty quickly.
In the interest of time,
I'm also going to kick off the integration
of the Data Warehouse.
So I'll hit Execute on that one as well.
Andrew Cress: And Andrew,
if it would help to maybe address
that question in the meantime.
We did have something from Bob in the chat.
And he's saying, can you remind us what's being ingested?
What data are being ingested here?
-: So the short version here is what we're ingesting
is the schema.
So we're pulling in information
about what tables are available,
what fields are available on those tables.
In some cases, depending on the data source,
and once we get in particular to Tableau,
we can show some of this off that there may be some comment,
information that's available
that can get pulled in automatically
if there's, again, some data sources,
if there's calculated fields.
There is in some data sources an option
to profile your data
which will look at some of the data in the tables
to give you some statistics around different fields,
how many rows there are, how many nulls there might be.
That is completely optional, and it's something
that is turned off in these recipes by default
so all you're getting is the schema.
The information in particular that these databases
are connecting to, they've got no data in them.
All we've done is we've populated the schema.
If you could actually poke around a little bit in here,
and if you're technically savvy,
you could actually get to the database itself.
You'll find that they're all empty tables.
All we're bringing in is the schema.
-: Okay, so now we've successfully imported
our fake Banner metadata and our Data Warehouse metadata.
So now, I'm going to click on the home icon
up at the top here.
And now, rather than that greeting screen
that said, "You're brand new here."
I actually have 79 datasets.
So one for each table, basically.
I'll click on that and then I can see in production
and MySQL for the type of database.
I now have two different databases,
so Banner and a Data Warehouse.
Let's start with Banner just so we can see
what our starting point is for some of us.
For those who are familiar
with the Student Information System,
you will see that every table name
is a helpful seven characters long,
and it's prefixed in DataHub by the name of the database.
So we called ours Banner, but in your campus,
it will probably be called something else
like BanPRD or something like that.
So we've imported around 40 odd tables from Banner.
This is by no means all of them, just to be clear,
there's thousands in most operational systems
but it gives us a good starting point for a demonstration.
Within any given table, you can click on it
and you can get the schema information.
So we can see at the top primary keys,
foreign keys, what type of field it is
and then we can work our way down the list.
If we want to, we can add descriptions by hovering
over the particular column and saying giving a description.
So we'll call this academic level or something like that.
On the right hand side, we can also add documentation
for the table itself as opposed to the individual fields,
or we can link it to external documentation
if that's applicable,
we can add tags to help us cluster like things together
and we can have a data owner
but Pat will talk about that a little bit later.
So that's our starting point on the Banner data.
If we go into the Data Warehouse data,
what we see here is that we prefix this with Data Warehouse
and we have information on anonymous students, applications,
course enrollments, course sections, et cetera, et cetera.
And it's the same kind of structure.
So here we've got schema,
it's got primary keys and foreign keys
and it tells us what kind of fields
the different fields are.
And then we can have table level documentation as well.
And we'll come back to some more stuff on this
once we're finished our next ingestion.
So if I go back to the home screen
and I click on Ingestion again,
I can now execute what we've called Lineage.
So in this case, we use a tool called FME,
which is made by Safe Software.
And it's a data integration tool that connects
to about 500 different platforms,
anything from locally hosted to cloud hosted, et cetera,
and everything in between.
It has the world's best support for spatial data,
and in the context of higher education
has a really compelling pricing model
for an enterprise data integration tool
that will allow you to connect
all your different systems together, automate the workflows,
make it so that when somebody uploads a file,
this can be reflected in your Data Warehouse
and ultimately in your visualizations.
It's a really cool tool.
So we highly recommend, we're happy to chat about it.
So I'm going to execute that one
and then also in the interest of time, I will...
Actually, that one should be pretty quick.
Any other questions in the chat that we've missed?
I think we got all of them. So for those who haven't seen,
if your lab says it's reconnecting or something went wrong,
this is "normal" at this point.
Just click the gear icon at the top
and go to the machine info.
To Julie's question, the name of the tool is FME.
At one point, this stood for feature manipulation engine.
Feature, being a field in your data.
Definitely worth checking out.
We've had some great universities and colleges
begin working with it in the last two years
since the higher ed pricing model became available.
And it's a visual tool to allow you
to integrate data from different systems.
Okay, so now our Lineage is available.
So this adds an additional layer here.
We now have a pipeline and a different platform for FME.
So out of the box, FME is not yet supported by DataHub.
We've built this custom,
but we imagine at some point in the future,
it'll become core to the platform.
So let's now search for the table STV term.
So this is Banner's term lookup table
that has information on what these different term codes mean
when the term start, when they end, et cetera, et cetera.
So now when I search this,
I can see a dataset called Banner.STVterm.
So let's look at that
and we can see here the different columns
that are available within that data.
But here I want to show the Lineage.
This is now available because we connected
our Banner database to our Data Warehouse
using that Lineage ingestion that read our FME workflows.
So now, let me just make this a little bit bigger.
So now I can see that STV term is used
by five different workflows from FME as a starting point.
I'm going to focus on stage application.
So stage application looks at application-like information
for say the current day,
and it's ultimately used further downstream
to compare that to what's already in the Data Warehouse.
Essentially to answer the question,
has this record been updated or not?
If it's been updated, I need to make a new record to say,
we have a new record starting today moving to infinity.
If it hasn't been updated, I don't need to do anything
and I don't need to waste my time
updating additional snapshots for example.
I'll hit the plus button, which was here
and to see where that leads.
So if I hover over stage application,
it highlights in blue information that's coming into it
and information that's going out of it.
The auto layout here isn't perfect
for presentation purposes so I will move this a little bit
so that we can see a little bit better
how these pieces relate together.
So now I can see my original STV term
feeds into stage application as does my term table
that I made in the Data Warehouse.
These feed in to a dataset called Application.
Those then feed into another staging table
that is the fact table.
So facts contain things like measures.
How many students applied as an example.
Last but not least in the chain for the Data Warehouse
is the fact application table,
which is ultimately what I would connect my Tableau
or other visualization tool to.
And so this part isn't live, but in a real environment,
there would be additional nodes to the right of this
that would show the connection directly to Tableau.
A question came in about learning management system data.
Those would be in the Data Warehouse
depending on the project,
but the basic answer is yes, they would be.
Last but not least here let's bring in Tableau.
So what this is doing is it's connecting
to our Tableau online instance
and it's looking at demo project for this particular talk.
And it's going to pull in the data source
and workbook information from that project,
and it's going to add those to what we have here in DataHub.
And that one's now successful
so I'll go back to the home screen again,
and now you can see we have additional categories
for dashboards and charts and platforms for Tableau.
So I'm going to start with the dashboard
and I'm going to go to Tableau and CAIR demo,
and then the demo workbook.
There's another one in there.
It's also fine.
And so we have a Fall Enrollment Summary dashboard,
and we can add table documentation to it
just as we could with database tables.
And we can add ch...
Or sorry, there are charts embedded
onto our dashboard already.
There's also a view in Tableau link,
but right now you won't all have access
to our Tableau online environment.
But if those was set up for you,
you could click that and get directly
to the dashboard as well.
So within this, we have a number of charts.
We can look at fall enrollment by college, for example,
and we can also look at Lineage
from the Tableau perspective,
remembering that ultimately this would connect backwards
to our Data Warehouse and eventually our Banner data
as appropriate, but it doesn't have access
to the same virtual machine,
so that part is a little bit smoke and mirrors for now.
So what we can see is we're looking at this one chart
that belongs to this dashboard and it really...
It uses a data source that in hindsight,
I should have renamed, but missed, called SCB course Banner.
It's actually about five tables joined together in Banner
and I started with SCB course,
so apologies for the renaming.
I can hit the plus button there
to see where does that come from.
And it turns out that there's a published data source
called Banner Course Enrollments that I did rename.
And so the difference between these two on the left
is the published data source is what I already published
a Tableau server.
The embedded data source is the one used by the workbook.
They're almost the same.
The difference is that there may be some calculated fields
in embedded data source that I never put
in the published data source.
So let's just view that
and then I will turn it over to Pat for the workshop.
Okay, this wasn't quite the screen
I was originally planning to show
but it'll come up in the workshop anyways.
So what I can see here is when I'm looking
at the fall enrollments by college,
and I click on the properties tab,
I can see which fields are actually being used
on my Tableau visualization
which can be really helpful when I go and try and figure out
if I change this field in my database,
is it going to have a knock on effect
further on down the chain?
What you can see here is that we have a term type field
that is a calculated field
and it pulled this directly from Tableau.
So our calculated field basically starts
at the case statement.
And the other part is a comment with a line break.
So I will search for the SCB course Banner,
just to find the same information
in the actual data embedded data source
which was the part I was trying to show earlier.
So here, we've got a number of different fields
in this data source.
Some of which are calculated
and here it better supports the rich text.
So if we scroll down a ways to find our term type,
we can now see that our formula
and the first line is a comment
and then our actual case statement is shown here.
So it's formatted a little better on this screen
than on the other one, but that's the rough information.
Okay, so we are at 12 o'clock.
So, I'm going to turn the screen over to my colleague Pat
for sharing the interactive portion of this demonstration.
So this is where we'll use the virtual machines
that you signed into previously.
If you haven't had a chance to sign in,
we'll post that in the chat again
if you're joining us a little bit closer to 12.
And this concludes the high level demo
of what is DataHub and what can it do for your campus.
So a couple of next steps just before people head off,
if you're interested in exploring a pilot with us
as design partners, let us know,
and we can talk about what that model might look like
and how your campus could get on board.
So this would be hosted by Plaid.
We'd take care of getting a set-up up and running,
pre-populating it with common SIS glossary type information
for things like Banner and PeopleSoft
and in the future, Workday.
We're there to help you.
We've used this before.
It is new but we've delved quite deeply into it
and we'd be happy to lend a hand as you get rolling with it.
We'll take care of testing and upgrades as well,
so I mentioned that they innovate really quickly
which means also that upgrades happen very frequently
and occasionally they need some additional help
to make sure that they're working,
so we take care of all of that.
And as came up in the chat earlier,
security infrastructure, so that basically,
nobody who's not authorized can access this.
So we'd be happy to help you along with this.
It's a new thing for everybody
and we can all learn a little bit together
by working collaboratively.
So reach out to us if you want to chat about that.
You can also go to the open source route.
So if you just want to get your hands dirty,
work on it with the programming,
that is totally fine as well.
There's great information documentation available
on the DataHub project website, it's linked there.
And then one other question came up in the chat
from Andrew earlier,
and he was wondering whether you need a ton
of database experience to get up and running with this.
And the answer I'd give with that
is that it would be helpful, but not required.
So at many institutions, the model is that IT
controls the infrastructure side of things,
manages the database, often builds these structures as well.
And the institutional research office
utilizes that information,
and maybe IT also manages something like Tableau server.
So your IT office could also set up say the Data Warehouse
and database side of this,
and they could be responsible for that part.
And maybe an IR you're responsible for things
like government reporting tables that IT knows less about.
It depends a little bit on your model of the institution,
but I wouldn't say that you need tons of database experience
to get up and running with this.
But what you probably would need
is knowledge of what information is available and where,
and that knowledge could be distributed
among different offices and you could have multiple users
in the same DataHub environment.
So I hope that helps a little bit, Andrew.
And then our contact info is on the bottom there.
Questions are welcome at any time.