Christopher Jordan: Okay, thanks everyone for joining us. This is our second session of kind of introducing and continuing the ISO bank ingest and now interface testing process.
Christopher Jordan: Several of you have already provided us with some sample data and everything. Everything thus far that we have tried to ingest the provided has we've been able to do so.
Christopher Jordan: Sometimes with some tweaks and needed and sometimes learning something in the process. So, so thank you especially to everybody who has sent us stuff already. It's been extremely helpful.
Christopher Jordan: I'm going to introduce today by talking just a little bit more about some of what we have some of what we've learned in some of what has changed as a result of the initial
Christopher Jordan: feedback that we've gotten from you all in the process of learning how to ingest data that that comes from the real world. And then I'm going to pass over to David walling
Christopher Jordan: Who's going to kind of walk you through the template generation and the ingest process as it is today. And that's going to act as kind of an introduction to to the process that we hope you will all
Christopher Jordan: I'll be able to use and test yourselves in the upcoming weeks and before I go into updates. I want to just take a moment to acknowledge the
Christopher Jordan: The fantastic work that that David and Thomas love and and have all done on putting together the the technical foundation
Christopher Jordan: For the metadata model and for actually inputting all of the metadata into the system, as many of you know we have a an extraordinarily
Christopher Jordan: Rich and robust metadata model with literally hundreds of of elements and we've come up with a system that will allow us to support a fairly flexible.
Christopher Jordan: Model with changing that data fields and changing controlled vocabularies over time without having to do significant reengineering but
Christopher Jordan: One of the consequences of that is that Anna has had to manually create all these metadata fields and be controlled vocabularies
Christopher Jordan: In the, in the interface and and she's, she's done a ton of work on that as Tomislav and David have done on the technical foundation so I'm really, I'm really grateful to have had them to do all of that because I, I certainly wasn't smart enough to come up with it so
Christopher Jordan: Okay, so a couple of things that we learned in the in the process of kind of ingesting
Christopher Jordan: The data that we've gotten from you already one of the first and kind of most important things we had initially been using a combination of the
Christopher Jordan: The user identity and the sample ID that the user provided as a way to uniquely identify both samples and measurements. We also had a lab sample ID.
Christopher Jordan: That was meant to be, you know, for a laboratory provided identity on the sample, but it kind of quickly became clear that we also had these cases where we have
Christopher Jordan: Multiple different analyses happening on the same sample. And so we need to be able to uniquely identify both a sample provided by a user and the
Christopher Jordan: multiple measurements that are associated with that sample. So as a result of that we we elevated. We already had an analysis identifier in the in the QA QC metadata, but we elevated that up to the core metadata and it's now a required field.
Christopher Jordan: So, you know, many, many folks I think will not have a an analysis ID that's provided by the lab, either because the lab didn't provide it or because the day that you're submitting
Christopher Jordan: Just you. You don't have access to that level of of meta data. And that's, that's fine. It doesn't have to be a really precise value. It just has to be something that is unique to that sample.
Christopher Jordan: So that we can, if necessary, differentiate different kinds of analyses that produce different kinds of measurements, but coming from the same sample that you provided to the lab.
Christopher Jordan: So that's a it's it's shouldn't be a very consequential change in terms of filling out and providing the data to us, but it's, it's something that you need to be aware of.
Christopher Jordan: Since you know it may, it may differ from your expectations of what the required fields are based on kind of previous discussions workshops
Christopher Jordan: The other kind of just minor thing we we have a number of fields in the place where this came up most frequently was the reference materials.
Christopher Jordan: For the lab and analysis metadata, where the the field can have multiple values in it. And rather than asking you to do something within a comma separated
Christopher Jordan: Spreadsheet to somehow differentiate multiple values within a comma separated spreadsheet. What we're asking folks to do is is just duplicate the column.
Christopher Jordan: And in that case, and provide us multiple columns with the primary reference material or whatever.
Christopher Jordan: Whatever field that may be, and we can handle that appropriately as long as we know that the metadata is something that we can have multiple values for that's something that the system supports. We have talked a little bit about
Christopher Jordan: Maybe providing more differentiation there. So, allowing for a primary reference material one primary reference material to and so on so forth. Because some systems.
Christopher Jordan: We've, we've heard that some of the data wrangling systems don't like having multiple columns with the same header on them. So that's kind of silly and I know been topic of discussion. But for now, that's, that's the guidance and that's that's the way that the system.
Christopher Jordan: Works with the data and that will hold true for for fields, the GPU generate through the, through the template generator
Christopher Jordan: And then finally, one something else I wanted to just highlight from the work we've done so far is we do have in the system for each of the
Christopher Jordan: metadata fields. We also have descriptive information we have descriptive information both for the metadata field itself. And in the case where there's a controlled vocabulary for each of the
Christopher Jordan: vocabulary terms within the list that we provide. And as long as we have those descriptions, we have a nice and
Christopher Jordan: If it's available. I think David will show you this. We have a nice kind of Field Guide to the metadata that will explain to you.
Christopher Jordan: What the
Christopher Jordan: What the valid values are what the metadata field is meant to be. And then details about all of the individual controlled vocabulary elements when we have them.
Christopher Jordan: The only consequence of that is right now we have, at best, very minimal descriptions for all of those
Christopher Jordan: That may come from you know from previous discussions in the workshops. But in a lot of cases, we don't have the
Christopher Jordan: Necessary expertise to provide appropriate descriptions so kind of just a heads up for everybody that we will
Christopher Jordan: We will be asking for the Community's help and providing useful descriptions for all of these various metadata fields.
Christopher Jordan: And I'll just say now if you see something as you're working with the system and you you have or no quickly how to write a couple of sentences that describe it do so and send it to us. We can put it in pretty quickly.
Christopher Jordan: If not, then we will be
Christopher Jordan: Organizing kind of a more systematic mechanism to get all of those all of those metadata descriptions populated because it is a very nice feature of the system. And I think it will be. I think it will be very helpful over the long term.
Christopher Jordan: In helping us data depositors to understand what exactly is meant by each of these metadata descriptions. What, what kind of data goes in each one of these fields.
Christopher Jordan: Because this is, as I mentioned before, the, the metadata model is is very complex. There's lots of terms and it's not
Christopher Jordan: It's not possible just from the descriptions of all of the fields to have perfect clarity as to exactly what it what we mean by every single metadata field.
Christopher Jordan: And how that matches up with your data that's kind of another another lesson. I'd say we've taken from the early deposit ingest process.
Christopher Jordan: Okay, so I'll just pause there. If there are, if there are any questions about any of that. I'm happy to take them. Otherwise, we can move on to to David in the demo.
Mariel Lee Campbell: I did have a quick question that I mentioned in my email earlier.
Mariel Lee Campbell: Today I'm sorry I came in late. This is Mario
Mariel Lee Campbell: Can up
Mariel Lee Campbell: Did you mention in earlier discussions, for example, where we would want to put scientific names or is that part of the description that you're just talking about
Text.
Christopher Jordan: Yeah, so, so the the version of the template that you were looking at was the earlier version that that definitely did not have tax on names.
Christopher Jordan: There is there is a space for that in the fuller metadata model and and i think i think you will probably see that, if not, if not today then then very soon.
Christopher Jordan: One thing I will mention about that is that for now. That's going to be a free text field we do intend to do validation against that, in the long term.
Christopher Jordan: But so far, we have good suggestions, but I wouldn't say we have kind of solid consensus as to how exactly we want to do validation of the
Christopher Jordan: The taxonomic names in that field. So we're going to kind of, we're going to kind of punt on it for the time being, and trust that you all mostly know what you're doing as far as inputting taxonomic names and it won't be too far off.
Mariel Lee Campbell: And then one other question in terms of locality.
Mariel Lee Campbell: We spoke about having some mitigate data guidelines for how to input data for these various fields and would be interesting to know like what level of locality. Are you assuming that we would want to include higher geography.
Mariel Lee Campbell: So total calories on the paper that I was putting data in for just said something like mainland, but I know that this is United States Alaska.
Mariel Lee Campbell: In a particular quad. And so I just wondered if there was plans were plans to try to do standardized locality information or if that would also be free text.
Christopher Jordan: He. I mean, for the time being, we're asking more for for coordinates than localities.
Christopher Jordan: The only thing without getting into the longer discussion. The only thing I'll say about about localities is that I would like to support it, but I think you have a similar issue there about word is your where does your validation come from what is, what is your source.
Christopher Jordan: You know, and, and we can we can rely on sources like Archos for a lot of this stuff, but
Christopher Jordan: You know, especially if we have multiple potential locality sources, then, then things get complicated so
Christopher Jordan: It's, it's definitely something I'd like to support, but it's it's a it's a more complicated topic. So for the, for the time being, I think we're more focused on just getting, you know, GPS coordinates, when we, when we can
Mariel Lee Campbell: Okay, great.
Christopher Jordan: Okay.
Christopher Jordan: Alright. So, David, I think you should have the ability to share your screen. Let me know if you need me to change anything.
Christopher Jordan: Okay. Thanks, Chris.
Can you guys hear me.
Christopher Jordan: Yes, we can hear you. Okay.
David Walling: Yeah. So here, we're still working and pushing all of our latest changes to a quality assurance or test website. And this is kind of where we want people to initially actually come in and use the site.
David Walling: This is something will keep around throughout the life of the project. So as we push out new features we can first have a
David Walling: Kind of a friendly user group test those out before we push them all the way up to production. So the URL is just ISO bank dash QA for quality assurance dot t ACC in texas.edu
David Walling: And we're very close to just having you guys come in and register for accounts and just to turn you loose on this website.
David Walling: There's a couple of things we want to do behind the scenes to finish up. That is, we want to make sure our backups are all set up to automatically take snapshots, even in the QA version so that if you do provide test data.
David Walling: We don't lose it. So we need a couple more days to get all that set up, but I did want to first go ahead and walk through very briefly how to register for an account and I so bank.
David Walling: So most of these pages will be open to the general public. So these first three tabs, for instance, our CMS generated
David Walling: public facing pages.
David Walling: Are data sets statistics page which we recently added will can also be something that's just available to the general public.
David Walling: But there's going to be certain features like my data that are actually going to require you to first log in, which of course means, you first need to have an account.
David Walling: So if you click Register or create account, you'll get a simple form, like you see on pretty much any website and I can just fill this out real quick.
David Walling: Once you hit register it should send you through an email your just your standard email confirmation step.
David Walling: You've opened that email accounts.
David Walling: Once you get that email, you should have one additional link to activate your account.
David Walling: Account activated and you can now login. So I'm actually going to use
Some news.
David Walling: So now if I try to access see my data, it'll force me to login for now I'm going to use this test account.
David Walling: And all I will say this in QA as you register for your account and start submitting data. The idea is least for this initial phase is we're going to try to move all of that data and user account information up to production as well.
David Walling: Eventually will be at a point where anything tested in QA will not be expected to stay around only the features and code will get pushed to cue to the production site. Once we get to that point.
When login here.
David Walling: So here is the main view for working with your data and the QA site for ISO bank. Here are three of the
David Walling: Sample Data Sets that we've received back so far.
David Walling: And here's where you basically can just manage all of these submitted data sets, which is the term that we use in the actual code, so refer to basically a CSV file.
David Walling: If I want to actually list all of the records associated with that CSV file. We have a separate page for doing that.
David Walling: This is highly editable. We can figure out exactly how much of the information to show on this page, but it's just a quick guide to let you see you know what records are actually associated with the CSV, I can actually sort on all of this, this information.
David Walling: And then for an individual record and by record, we really mean a line in a CSV file, you can click on the details page for it. And this is kind of the more extensive
David Walling: Information about that particular line in a submitted CSV file. So in this case, this particular line only had a single measurement type of detail. Ah.
David Walling: But this particular value and an unknown for the unit, but it did provide the scale information.
David Walling: These are the measurement specific metadata fields. The rest of the additional metadata is all of the metadata that is shared by all the measurement types for a given line.
David Walling: In a CSV file. And for each of those we are grouping them into certain categories, just to make the management of them easier, since we do have so much information and for any given
David Walling: Record. There may be
David Walling: You know, maybe a single value, maybe 50 values and this page will be dynamically created based on the actual metadata values that are present for this particular record.
David Walling: You can also at any time download a CSV, the actual CSV first version of the file that was submitted and this gets back to how we expect updates to these files to work.
David Walling: Well, as I mentioned that the last time we want Excel or whatever your favorite CSV editor.
David Walling: Is to be that primary editing interface because it already does this very well it's scripted, you could be. I think we already have one user is actually using our to spit out and manage kind of all these
David Walling: CSV files. So we want y'all to keep using the tools that you're already comfortable using. We don't want to try to refill recreate all this type of basic functionality within the application itself.
David Walling: So you could come in here update this this information fix them values add to it, whatever. And then you would simply resubmit that now, right now we don't have an update button.
David Walling: So for now, we could simply delete the file.
David Walling: And then just resubmit it
David Walling: We are requiring there to be a unique title for each one of these files you submit, as well as providing some sort of description just to help you and others who come and looking for data kind of understand what this is.
Choose our file.
Which one actually deleted me double check.
David Walling: Go back to my data.
David Walling: Recreate
David Walling: And once I submit, you'll see the button is grayed out. So I can't hit it again. It's good, it's depending on the size of the file and the number of metadata fields. It may take anywhere from five to 30 seconds to actually load all that data.
David Walling: For each line it's you know it's verifying all these
David Walling: Information about the metadata fields, whether it's required, whether it's required only if another field is present on that particular line, so on and so forth.
David Walling: But at the end of that if it successfully was able to create every single line and that CSV file. It'll take you to the summary page which tells you how many records were actually created
David Walling: And then for each of those that actually give you a list of warnings. So these were things that did not cause the file to be rejected that it's the same thing that we want you to be aware of. So for instance,
David Walling: Each row and this particular CSV has a column called external sample ID, which is not matching up to a known column name for a supported metadata field.
David Walling: So this is a feature we wanted to have this means that you can actually include additional columns in your CSV, that we don't yet support and at some point in the future we may add a metadata field that matches that column name, essentially, and you could simply, you know,
David Walling: resubmit that or eventually we'll have an update field. So you can hit Update pass it along the
David Walling: That data CSV file, instead of just deleting all the records will actually update them on the fly.
David Walling: This will become more important as we actually build additional functionality on top of these submitted data sets. So for instance, maybe as we start sharing these between users and projects.
David Walling: You may want to keep a list of your favorite sets. So in that those cases, we're not going to want to just completely delete the data set and recreate it because we're going to have foreign key References
David Walling: Within this kind of favorites, a concept that we want to keep in the database. So the actual technical details behind that is actually not that difficult. It's just we have something we haven't focused on
Fight at this point.
David Walling: We can see that so so in QA right now, these are the only three data sets. We currently have loaded
David Walling: But if we look at our data set statistics page, we see that even with these three we have 267 unique analysis records.
David Walling: Almost 4000 unique analysis metadata records associated with those 658 measurement records and for those 658 measurement records 790
David Walling: Measurement metadata records, not every measurement type has measurement metadata associated with it. So things like percent carbon or just a value. There are no real metadata records, at least at this time directly associated with that with that measurement tight.
David Walling: But we we expect to fully fleshed this page out with some other interesting things and we are open to suggestions.
David Walling: About what should be here. You can imagine the table that shows counts by measurement tight.
David Walling: Counts by analysis type material type, so on and so forth. So this is, again, kind of a page that we can use to market ISO bank eventually to the wider community and it can be accessible to people who haven't even logged into the system.
David Walling: Any questions about this kind of, oh, so I didn't want to show. So what happens if you try to submit a data set that has an error.
David Walling: So if I select one, let's say,
David Walling: This one, which I have some known errors in and I tried to submit that the feedback mechanism is it will if there's any line in that CSV that has an error.
David Walling: It's going to reject the entire file, but also give you a row by row information about what was wrong with that CSV. So in this one.
David Walling: We saw that no one lab sample idea was not provided with that's required field and also enrolled one we had this d 13 see measurement scale field that had a
David Walling: Value of bit bad so I did not match a lookup value. So measurement scale is a lookup value metadata filter. So the value in that if there is a value.
David Walling: It would try to look that up in the database if it can't find it, it will reject it.
David Walling: So on and so forth. So the feedback mechanism is you'll you'll bring your data in, there's going to be based on one of these upload templates which will dive into next
David Walling: And you'll fill that out will come here try to submit it. If there's errors. We want this to provide enough information to the user that they'll be able to go back and fix those errors in the CSV file and resubmit that
David Walling: Any questions on that before I jump into actually, how do we create an upload template to start filling that out.
Okay.
David Walling: So we showed this off at our last meeting, but we've done a little bit of work to kind of hardening it and a few minor enhancements as well.
David Walling: The first thing we'll see here is that we have added this reset form button. So as you're going through and clicking these. There are so many of them that
David Walling: You may get to a point where you're like, Okay, well I just kind of need to start over. So by hitting reset form, it's basically going to
David Walling: Pre select all the required and recommended metadata fields as they're currently specified in the system. This is also the same mechanism that we expect to use to provide community specific metadata templates. So imagine another button that says water or paleo, we could
David Walling: Have that community decide what are the most important or what are the metadata fields that they really want to be in all
David Walling: data sets that are uploaded and associated with that particular community. And so we can have those fields to click on a button, have all of those particular fields pre selected and then at the bottom, allow them to download that template, fill it out and submit it to us.
David Walling: I think the I think since last time. We've also added this measurement types area. So you have to select at least one of these measurement types in order to download the template.
David Walling: If you try to come down here without doing that, you'll see that the selected measurement types is empty and this button is basically turned off with a message to please select the measurement type
David Walling: So if I come back up. Select the comments, carbon, nitrogen fields. They now show up in the selected measurement types and this button has activated.
David Walling: We do have
David Walling: Within the system relationships between certain fields, trying to remember exactly which ones we've been testing on, I think it was
Tomislav Urban: Click today. I always like to use soil in material type and then soil horizon.
Tomislav Urban: As a dependent field. I hope that
Tomislav Urban: QA. I know it is
Tomislav Urban: On my database, but I saw
Tomislav Urban: A note to try sorry the inorganic version of soil scroll down a little bit more
Tomislav Urban: There we go. That one.
Sorry.
Tomislav Urban: Yeah.
Tomislav Urban: Looks like it's not set up in QA
David Walling: So this is a lot of JavaScript stuff that we're still kind of working through some of the bugs. But things like
David Walling: Blinking now and which ones will trigger the there's a lot more actually in QA than there was even here.
David Walling: Well, anyway, the point is that some of these by by clicking one if there's a relationship that exists, it will automatically select one that's dependent on that. So if I select
David Walling: One of these fields and another one is required if that fill the selected it should automatically implement that logic that can also that can be true.
David Walling: With both the just the metadata field, but also certain valid values within a metadata field could trigger a relationship.
Anna Dabrowski: That says that
David Walling: An additional fields should be provided. If this metadata field has a certain value. So that's a lot of what we've been focused on over the last couple of weeks.
David Walling: But nevertheless. Once you've selected your fields, you should be able to hit the download button.
David Walling: Spit out your CSV, open it in your favorite CSV editor.
David Walling: And here we see the all the fields. The main thing I wanted to point out about this and then I don't think that we've really talked about
David Walling: Previously, is that there's a certain order to how we're putting these spills into this, the CSV files. Basically, it's going to be the core metadata fields.
David Walling: For each group of analysis metadata will provide those columns. And then at the end.
David Walling: For each measurement type, we will provide a column for that measurement tight and if that measurement type has
David Walling: metadata associated with it. Those will always follow directly after that the measurement type. So this is done for two reasons. The first is we want to keep these templates and the CSV files that users will download from ISO make
David Walling: as consistent as possible. So it just kind of helps you. If you're looking for a particular value you kind of start to understand where it would be within the CSV files.
David Walling: This is ordering also provides a way to simplify the ingest logic slightly because there is an implicit relationship between
David Walling: Well, it's technical, but if I have like all the court measurement all the core analysis data, I can actually create that analysis objects and then start tacking on all the analysis metadata to it.
David Walling: And so for those two reasons we are at least at this time making the order of these columns have actual meaning.
Mariel Lee Campbell: I have a question regarding that. So if you wanted to add a new field or column you would need to go back and download a new template.
Mariel Lee Campbell: And rather than just inserting the column with the appropriate header.
David Walling: somewhere and you're fine. Actually, you could do both, actually, it would be okay.
David Walling: So if it was a new analysis level metadata. You could just tack it on you. It's probably wouldn't break the ingest. But again, we do want to try to keep them as
consistent as possible.
David Walling: So you could do it both ways to do this. These don't have to come from the download tool or the generator tool you could
David Walling: Create your spill it out and then hand it to a partner is like an example of doing and have them fill out their own version of that just based on the CSV file we do anticipate that these things could just be passed around as blank templates or you know header or CSV files.
Mariel Lee Campbell: But the order would need to be the same as it would be if it were you downloaded the headers. You couldn't just depend or add in a column in some random place within the file.
David Walling: That's a good point. It may break it. There's, there's, each individual
David Walling: One doesn't have to be the exact same order. It's more the groups of them right. It's more like these first four need to be the need to be in some order at the start.
David Walling: All the analysis metadata needs to come in the middle, followed by the measurement stuff at the end. You could if it's an analysis measurement field, it could go anywhere within this middle group.
David Walling: But that's actually a good point that I'll have to kind of think about the implications of how that might complicated and just
Christopher Jordan: Yeah, I think the the point the point there would just be, you know, if you if you know roughly where it goes and exactly what it's called.
Christopher Jordan: It's probably okay to just add it to the spreadsheet, but but there's not going to be any guarantee if if you are just kind of randomly sticking new fields into the middle of the spreadsheet that didn't work the way you expect it to.
Philip Manlick: The follow up on that, Chris.
Philip Manlick: Oh, sorry, this is Phil, I was just gonna ask if we could shuffle things around, right. So I think like these are reports that UNAM when we always give them the people I'm thinking about an outside user
Philip Manlick: It'll say, you know, D 13 see percent CD 15 and percent and then our carbon and nitrogen ratio. And if somebody wants to just copy and paste
Philip Manlick: That chunk of their report and put it in. Could they reshuffle these headers to make them the order of the spreadsheet that they already
Christopher Jordan: Know that's that's what that's what David is saying this order is important. There are, there can be variations within the groups but but if you if you significantly change the order, you're, you're definitely going to a point where it's no longer necessarily guaranteed to work.
David Walling: Yeah, if we could technically make it work in any column order, we can handle all the reordering on the fly, but I keep advocating to the tech folks that I think trying to keep these things as consistent as possible is going to be a good thing.
Mariel Lee Campbell: I think that having them consistent as possible on the doubt template download is really helpful to users. However, I see a problem with multiple users or even one user trying to
Mariel Lee Campbell: Edit a file as most people are used to being able to shuffle their column headers around that that doesn't really matter. And if you want to add information later and having to constantly go back and follow some we would need really clear.
Mariel Lee Campbell: Some clear interface instructions as to what the appropriate order should be to avoid problems down the line when people are actually working with the spreadsheets.
Philip Manlick: I agree with Larry. I think that's pretty common that people are going to want to move these things around and at least putting it on the page.
Philip Manlick: You know when you're creating your template that it needs to keep in that order, so people know because I think that a logical things people might
Philip Manlick: Do is reshuffle these headers to match this spreadsheets that they already have, because it's easier to do that then data but you know spreadsheet that's already full of data.
Mariel Lee Campbell: Yeah, cuz I mean just doing the test file.
Mariel Lee Campbell: From the last time around I realized, Oh, well I could add a publication.
Mariel Lee Campbell: In July, and do this. And oh, okay, I'm just gonna, I'm just gonna grab these two columns. I'm going to insert some new information and I just didn't put it wherever it felt like it made sense but I
Mariel Lee Campbell: Know, I would want to have the flexibility that most people working with spreadsheets are used to that kind of flexibility and I think if we didn't have it there would need to be very, very explicit instructions and it might create some user difficulties.
Mariel Lee Campbell: Okay.
David Walling: Well, I
David Walling: I would propose this let's make the ingest column order agnostic, but I would, however, when you hit this download button.
David Walling: I would have it actually store the CSV file in a particular order, so it can interested in any order. But when it's actually saved and insert a backup. It should be in a more consistent folder semi
Mariel Lee Campbell: I'd be great. Yeah, that's fine.
Philip Manlick: That would be great. Yeah, it'd be awesome.
Christopher Jordan: I think that that's technically not teaching so
David Walling: I will explore that.
David Walling: Right. So we took this last time as well. I think that we have this. So in addition to this kind of forum where you're actually selecting which field you're actually, once we also have a
David Walling: More in depth field guide which it
David Walling: opened up a new tab here it's going to show you your core analysis information. And for each of the metadata fields or core analysis information we
David Walling: Provide the column name, whether it's required a little text about, you know what, what is this. So for lab, for example, it's the abbreviation for lab where the analysis was run and unique to lab, is that a matching laboratory must already exist in isolation.
David Walling: We want lab to be like a first class entity with an ISO bank and so as we're starting to test and we're getting more labs in that lab likely may not already exist an ISO fake so
David Walling: We want to talk about how to we don't want any lab, just to be entered by anybody right we want some kind of control over that. And we're still debating how we should handle that. So for now, you can just send a
David Walling: An email to the ISO bake attack email list. If you get an error saying this lab doesn't exist just send us the information about the lab on it as an abbreviation that it's commonly known by give us that, and we'll go enter in through the admin form and have that available for your ingest
David Walling: Sorry, that the field guide lab sample ID.
David Walling: Again, it's required some of these fields are going to have
David Walling: reg ex or other validators on the lab sample ID. It's free text that we want you to keep it between five and 50 characters and then only alpha numeric values with no spaces.
David Walling: So we also provide some examples of what that looks like. Same for user sample ID same type of requirements on the field. And what's a valid value for that field.
David Walling: What that field actually represents, and this is the sample this simple measurement measurement ID is the field that Chris pointed out at the beginning of the meeting.
David Walling: That is our kind of new unique ID that essentially matches and uniquely identifies a line in a CSV file across actually all of ISO think right now, not just a given.
David Walling: So it's five Charmin 50 char max alphanumeric no spaces in it. Again, this key must be unique for every record submitted to isolate
David Walling: And you, there's the user can kind of come up with your own kind of unique formula for that. So, for instance, for this one.
David Walling: We could just easily and this is easy to do an Excel or anything else just tack on a one or two, whatever.
David Walling: If there's duplicate record say in your user sample it to easily create this unique sample mission and ID. Once we get to the point of updating records and as opposed to deleting and recreating will be king off of this particular field.
To make those updates.
David Walling: It's pretty important point. So any questions about that or confusion about what that represents
Mariel Lee Campbell: Haven't you have a quick question question, David, is it
Mariel Lee Campbell: Some of these ideas are going to differ from, for example, lab sample idea. You said it's alphanumeric no spaces, the data said I was provided actually
Mariel Lee Campbell: Did have spaces and so you'll be changing the original sample ideas, it was
Mariel Lee Campbell: Run in lab but course. Ideally, in the future, people will have some understanding of the standardization here.
Mariel Lee Campbell: The same thing with sample measurement ID, is there a possibility that you would auto generate some of these values rather than putting the recording the user to do so to make it stand right
David Walling: we've kind of gone back and forth on this. We talked about this using the primary key in the analysis table for this. Um, what
David Walling: I guess we haven't come to a complete consensus, this isn't working for us in the past few weeks, and I guess it's
David Walling: We're still considering open for debate. Chris, how we want to have on this.
Christopher Jordan: Yeah i mean the the fundamental issue. You know, I thought about uniquely about, you know, just auto generating these in cases where
Christopher Jordan: Where they're not supplied, but the, the issue is that this is an important field that is that is used to uniquely identify things so we don't at the point of ingest we don't have enough information to say that auto generating it is the correct thing to do.
David Walling: A good point, actually.
Christopher Jordan: So, Linda provide us to measurements right for the same sample and just omit a measurement ID and one and then not omit it in the next one.
Christopher Jordan: And we could do the wrong thing. Basically, in process and parsing through the file. So, for the time being.
Christopher Jordan: We're making this a required field and you'll see what what David has provided here is an example where you're basically just copying.
Christopher Jordan: You're just copying a user sample ID into the sample measurement ID. And that's, that's just to tell the system. Yes, this is it. This is a unique measurement
Christopher Jordan: That is associated with this particular sample, you know, we can talk in the future about if this is a case that we see a lot. Maybe there's a way to tell the ingest system. No, absolutely. I know what I'm doing. I want you to generate these for me.
Christopher Jordan: But, but that's about the best we could do. We're never going to have the information to be confident that it's okay for us to you to generate those for you unless you explicitly tell us that that's the case.
Mariel Lee Campbell: So just looking at your neon numbers right there. For example, having worked with neon and having just looked at them to a different researchers samples.
Mariel Lee Campbell: Neon is pretty good about their
Mariel Lee Campbell: Their formatting and it's pretty consistent in their way they do things but independent researchers are not the file. I was just working with head. Sometimes, there was a stat.
Mariel Lee Campbell: Extra Space between a number and a letter and sometimes there wasn't. And it was and I could see problems with user error, causing a lot of conflicting values, we have the same issue in museum collections with coming up with global unique identifiers and I can see that.
Mariel Lee Campbell: Kind of being an issue here that there will be error people submitting data will give you very multiple versions of the same thing. I don't know how much of it, you can control.
Mariel Lee Campbell: But having one standard key that was assigned uniquely within is a bank that was outside of the control of users to create their own error might not be a bad idea, at least for the critical fields.
Christopher Jordan: Yeah, and we will be clear, will we will have that you see on this, the far left hand side, this ID that is a unique to ISO bank ID that we generate
Christopher Jordan: That that that uniquely identifies the values within ice. So we do have that we just need these fields so that so that people can, for example, give us multiple measurements that relate to the same sample and identify that that's
Mariel Lee Campbell: What about
Mariel Lee Campbell: Later correction or
Mariel Lee Campbell: To any of these the same problem is a gen bake has the same problem. If a submitter puts in, for example, the wrong about your catalog number
Mariel Lee Campbell: Or it's in the wrong format or it's just blatantly wrong.
Mariel Lee Campbell: Then, and it's associating then that sample with the wrong museum voucher specimen.
Mariel Lee Campbell: As a result, then it's only the submitter can make those corrections usually don't have the ability for museums to make corrections later. But there are needs. There have been issues where we do need to make corrections to associate evaluate the right samples. So I've just kind of
Mariel Lee Campbell: From, from my perspective, working with Jim bank and with museums both
Mariel Lee Campbell: Just heads up that there will be there will be problems down the line. And the more you is you know you have unique key on something. It just helps resolve those issues.
Yeah.
David Walling: Second session section is the core measurement information right now. That really means measurement type and then everything else is mostly
David Walling: The analysis and measurement so meta data fields grouped into their categories for each of these. If you look at if you click the Help button, it'll pull up a
David Walling: Additional page.
David Walling: With a little easier to read formatting and this is where we can provide this more detailed description that could be quite
David Walling: verbose. If it needed to be for Turkey Phil and additionally for the valid values could also have their own descriptions for each of those valid.us as I think Chris Thomas mentioned earlier.
David Walling: And again, this is where we would be looking to the community to provide us or actually enter and help us manage this type of metadata information about the metadata.
David Walling: I believe
David Walling: Yeah. So as we turn you loose you more than likely in both with the upload templates.
David Walling: Or actually trying to submit filled out templates run into errors. And so for now we're just going to ask you to provide again to the
David Walling: Lists that I saw big attack that you've texted a to just shoot us an email with as much information about that area as you can so we can go track down exactly what's going on.
David Walling: And just all use that as our primary feedback mechanism right now. Eventually, maybe we'll have some more kind of formal
David Walling: Log a bug report type functionality, but as our community is still relatively small, we'll just keep a little more informal and flexible in that regard and Thomas Otter curses anything that I've missed that we wanted to cover for this particular
Christopher Jordan: Will just on that, on that point. If you are if you're reporting something that you think is a is a bug that is you're getting an error on ingest and you believe you know
Christopher Jordan: This shouldn't be giving me an error, then yes, please do report it to us. But ideally send us send us your CSV file as well.
David Walling: Yes.
Christopher Jordan: So that we can replicate the bug and and verify that it's fixed if it is and and I definitely do encourage you to do that. Try this stuff out, it will break. Don't worry about that will definitely break
Christopher Jordan: But that's fine. That's, that's why we're doing this just, just let us know how you, how you think it's breaking incorrectly and give us the the CSV, so that we can validate the behavior when it's fixed
Christopher Jordan: Okay. Any other, any other questions or comments on on any of that.
Christopher Jordan: So yeah, as, as David said, we have a kind of a couple of other things to do just basically on on hardening the server. So we'll
Christopher Jordan: We'll send you a follow up message within the next few days to let you know it's it's good to go. I won't necessarily discourage you from playing around with it as it as it is.
Christopher Jordan: But, but just be aware that things, things may be changing quickly and coming up and down over the next couple of days. But certainly, you know, feel free to play around with things.
Christopher Jordan: Look at the Field Guide. Again, I'll reiterate, if you see
Christopher Jordan: A description there that you know we put in a minimal description and you think I have a much better description in this field, type it up and send it to us where we're happy to take all of the feedback we can get on that for the, for the time being.
Christopher Jordan: If there's nothing else, then we can let you all go I did record the session. So I will, I will send out links to all of that for those of us who couldn't make it today.
Christopher Jordan: Alright, thanks everyone.
GABRIEL BOWEN: Thank you guys.