Web-based responsive document annotation tool
| Name | Summary |
|---|---|
Statistics about the accuracy and IAA
Intermediary interface which marks subtypes as elements altering the state of the application by setting some annotation value
Analyzed individual annotation result
A document with additional statistics like agreement and correctness
Request body for analyze endpoint
Analyze response with TopLevelStatistics and a list of AnalyzedDocuments
View data class - all properties relevant to annotate / curate a project in the frontend
Base class for any annotation. An Annotation is defined as the actual value associated with an AnnotationDefinition. For example, a BooleanAnnotationDefinition will produce a single annotation of boolean type. This annotation might be created by a human annotator, or automatically generated or imported. Every Annotation needs to be associated with an annotation ID / key in a map or comparable structure
Base class for all AnnotationDefinitions. An AnnotationDefinition defines how an annotation is required to be created, e.g. on which target it is defined, is it optional, maximum length etc.
Wrapper class to prevent type erasure of AnnotationDefinition. This way, the JsonSubType info is preserved.
Model for annotating a document for the frontend
A single EnableCondition applied result, indicating if an annotation is required or not.
Base class for AnnotationGenerators. An annotation generator is responsible for automatically generating an annotation value, optionally with a probability attached, for a specific annotation definition. This can be used for statistical methods, simple if-else annotation (for example: if value in list of values, annotate with X), or for machine learning integration.
DAO for AnnotationGenerators
Data class representing a single annotation result by either an annotator, a curator or a merge of annotator responses by the policy logic.
The AnnotationResultCreator represents by whom an AnnotationResult was created, which can be by humans (Annotator, Curator), a machine Generators, externally Import or through consensus of all the previous Consensus.
Equivalent to AnnotationResultCreator but streamlined to a single displayName for the frontend
Return model for storing endpoints, letting the frontend know the data was stored successfully. If not, will return validationErrors
Annotation schema as defined in the database (with ID references to AnnotationDefinitions and AnnotationGenerators.
Single element of the schema, mapping the ID of an AnnotationDefinition to data specific to this project and the Target of the annotation. different target definition for the same annotation
Get some data from some annotations
Base class for subclasses that define how some data from a Document or GeneratedAnnotationData is extracted and used for EnableCondition or annotation generation.
| (extensions in package common) |
| (extensions in package application) |
| (extensions in package common) |
To have easy access to the configuration properties of the application, this class exists
AtomicEnableConditions are defined in relation to a single referenceKey and do not include more complex conditions such as And or Or.
Element displaying a constant Base64 encoded image
Display a meta data element that is a base64 encoded image
Wrapper element, all children’s text elements will be bold. Equivalent to an inline html element with fontWeight bold which will be applied to the children.
Annotation for a boolean input.
How to handle differences in cases between tags?
Data class to receive a check enable condition request
Data class for the response body of the check enable condition request
Annotation for a closed number with a min, max and required step. Necessary if you want to display annotation as a slider.
A column is part of a row and has a width dependent of screen size. Every row should have at least one column.
The UI defines 5 different screen size breakpoints, from smallest (xs) to largest (xl). The actual pixel breakpoints are defined by the UI. Column sizes should be in 1,12 range, 12 being the full width of the row, 1 being 1/12 width of the row. At least xs needs to be defined.
Equivalent to And connection between Contains conditions, exists as shorthand Example: { tags: { $all: ssl,security } } => { $and: {tags:ssl},{tags:security} }
| (extensions in package common) |
Data class representing the CORS config
Use this to define a non-text index with an order as well as an optional unique constraint
Filter based on a mapping a date string to timestamp and then performing GTE on the value provided.
Filter based on a mapping a date string to timestamp and then performing LTE on the value provided. If the value is not existing, won’t return the document
Convert some input date format (e.g. timestamp) to date format or do nothing if not a timestamp for a given meta data element
How should the date range filter work
| (extensions in package common) |
Annotation schema as sent to the frontend / client, contains the actual models of AnnotationDefinition, AnnotationGenerator etc.
Equivalent to AnnotationSchemaElement but denormalized, meaning all ID references are replaced by the actual objects
AnnotationGenerator which detects the language using the “Lingua” library, storing the annotation under the defined key as the ISO 639-1 Code (in upper case). The “unknown” key from Lingua is defined as the string “UNKNOWN”
Intermediate interface specify that an Element is non-interactive, allows for restrictions of classes to require Elements to be non-interactive
The data class representing a document with some originalDocument, a unique id, optionally a restrictedProjectID (marking the document to belong only to that one project), and the projectAnnotationData holding all the annotations for every project related to the document.
This DAO provides all methods to interact with the document collection. It hides the collection and controls access to it via the public methods.
To make code more readable, define the ID of a document as a typealias on String
Given annotators and curators the ability to further specify which documents are shown to them
Equivalent to DocumentSelection but with aggregated options inside
Individual statistics for a document
Use this for annotations that should be created for the whole document.
An annotation which is targeted on the whole document, not a specific part of it. For example, a class label “SPAM” or “NO SPAM” for the whole document.
Element displaying the document text.
| (extensions in package common) |
AnnotationSchemaElements can be conditional, defined by an EnableCondition. If an EnableCondition is null, that means it is always required. Else, the enable conditions execute method needs to return true for the element to be prompted.
Use for direct equals or array contains
Enum representing detailed errors which are more granular than just the http status code
Export configuration for consuming the results of this service
What aspects for the document and results to export
Data class holding information about where and how often the annotations were exported.
Light wrapper classes around MongoDB query elements. To use the FilterCondition as a parameter for MongoDB find, call buildQuery +on the FilterCondition object. Support for:
Condition system which decides if a generated annotation can be finalized automatically
Data class representing a finalized annotation, which can be one or multiple annotations (referenced by their IDs). It also contains meta data about why the document was finalized, which policy was used, when the finalization happened, and statistics about how the annotations were exported.
Finalized annotation result used for the analze response
Sealed class with two options why a annotation can be finalized
Custom exception indicating the user is not authorized properly (missing role), will be used to return appropriate http error code
When a Project has any AnnotationGenerator defined through the AnnotationSchema, they will store their results here. Every time annotation generation is triggered, every generator will be executed again and a new instance of this class will be added to the ProjectAnnotationData
How to handle sorting of documents for annotators when generated results is available
Exception indicating that a resource is gone, will be used through StatusPages feature to return the proper status code when this exception is thrown
Annotation definition which requests the annotator to chose from a set of predefined TagSetOptions
Inspiration: https://www.sparkpost.com/docs/tech-resources/webhook-authentication/
Base exception for HTTP errors, can optionally include a detailed http error
Data class to return on an http error, which can have a more detailed errorCode explaining the reason for the status code
| (extensions in package common) |
Display a material icon
Return list of generated documentIDs in order of imported jsons from request body
Request body to import annotations
Mapping of originalDocument to documentData used in UI. Will be mapped based on this classes values. Additionally, provides the ability to define indices for faster queries in DB
Data class representing log data from the interaction of the user with the document during annotation.
Wrapper element, all children’s text elements will be italic. Equivalent to an inline html element with italic font style which will be applied to the children.
| (extensions in package common) |
| (extensions in package common) |
Data class representing the JTW configuration
Data class representing the JWT validation configuration
Data class representing the HTTPS config for ktor
Define the layout of how annotation interactions / inputs will be displayed in the UI.
Mapping of layout areas to list of rows containing UI elements
There are 4 different UI areas which are defined by this enum. All four areas can contain read-only/display elements, but not all are allowed to have all types of interaction elements.
Interface for all UI elements, uses json polymorphic deserialization to map into actual UI elements.
| (extensions in package api.manage) |
View data class - all properties necessary to display project in list in the frontend
View data class - all properties necessary to display project in list in the frontend for managers
View data class - all properties necessary to display and edit project from management perspective in frontend
| (extensions in package annotationdefinition.generator.documenttarget) |
A message is a text between two users, optionally relating to an AnnotationResult
DAO for the Message model, controlling access to the message collection.
For a meta data element, use the value as a key to the mapping map and display a list of DisplayElements or the fallback if no value is found for the key
Data class representing the MongoDB config
Wrapper element, all children’s text elements will use a monospace font. Equivalent to an inline html element with a monospace font which will be applied to the children.
| (extensions in package common) |
| (extensions in package api.annotate.dto) |
| (extensions in package common) |
A number range between min and max with step steps between. Results in two values, a lower and upper value.
| (extensions in package common) |
Define behavior what to do when a finalizedAnnotation was defined, but a new one was set afterwards. This would not happen normally, but cannot be prevented for cases where an annotation is found via search to correct an earlier mistake. For this case, we need to define if we want to trigger web hooks again or not.
What to do when calling the WebHook failed
Annotation for an unrestricted number. Cannot be displayed as a slider, only number input.
HTML Number input
Tags annotation with the ability to add new tags from the user. Here, the tag is just a string / the value is the actual string
Chips element is a variable list of text inputs (tags) which can be extended, including an auto-complete feature with predefined answers.
Annotation for some open text input
A multi line text input field
Enum representing the order options, ASC and DESC
Get some data from the originalDocument of a Document
A Page represents a UI page of the frontend, optionally with a badgeCount to indicate how many interactions are waiting for the user
Model for frontend communication representing the PageSetup, controlling the layout and core data for the page setup of the frontend
Wrap the percent by storing n, the absolute value as well as the calulcated percent value.
| (extensions in package common) |
Data class defining policy of how to handle documents / annotations, especially how and when to finalize an annotation for a document and project.
Sealed class for the different actions that can be required to be taken for a document to get the annotation done properly
A popover element, providing the ability to hide information (no interactions!) befind a popover
The PopoverContent are display elements which will be shown when the popover is visible
The PopoverTarget will be displayed into the parent context directly (for example some text, an icon or a combination)
Two ways a popover can be triggered, by click or hover
Data class for annotation and curation endpoints for receiving annotation results. Will be mapped into other data structures for storing.
Complete model of a project. This model represents the database structure / is stored in mongoDB
Data class representing all annotation data for a specific project.
DAO for the Project regulating access to the project collection
Response body for storing a Project with success info and validation errors
A single validation error for a specific key of the ManageProject
Information about an export via rest call, mainly the route called, how often, and when.
What REST endpoints to activate and how to export them
| (extensions in package api.admin) |
| (extensions in package api.annotate) |
| (extensions in package api.export) |
| (extensions in package api.generators) |
| (extensions in package api.import) |
| (extensions in package api.manage) |
| (extensions in package api.pagesetup) |
| (extensions in package api.search) |
| (extensions in package common) |
Equivalent to a Row of UI layout systems like Bootstrap or Material UI
Model representing a search request
The SearchResultDocument is a view on a document with all useful data for display in the search result area in the frontend.
How to select a subfilter
Data class representing mongo DB sort
Single sort element with key and order
A span is a single piece of a string, expressed by begin and end.
What is the allowed cutoff point for a span, a single character or only whole tokens?
Use this for annotations that should be created for a specific span. Define characteristics of how the span should be set.
All annotation values for a single AnnotationDefinition on a span target are represented by a SpanTargetAnnotation. For example, a document might have spans of positive and negative sentiment. All annotations regarding sentiment would be stored in a single SpanTargetAnnotation, holding a list of annotations of which each defines the spans concering the annotation as well as the actual value, in this case POSITIVE or NEGATIVE sentiment.
A SpanTargetSingleAnnotation maps a single annotation value to the spans (often only a single span) which the annotation value is associated with.
| (extensions in package common) |
Class to compare value inside document and value both as strings, using Mongos expr feature
A SubFilter is one element by which a user can filter documents shown
Option aggregated with count
For subfilter selection query to map the result
A subfilter but with options aggregated
Annotation definition which requests the annotator to chose from a set of predefined TagSetOptions
A ButtonGroup is a collection of buttons which belong together. Based on the referenceAnnotation it can be single select or multi select.
Use this class for external ML models which are already trained and don’t need any training data
Generic updatable AnnotationGenerator for TagSetAnnotationDefinition and DocumentTarget.
The target of an annotation can be the whole document or a specific offset of characters, a span.
Enum class representing the two Target types, equivalent value to the JsonSubType of Target class
Just display some static text
Display the value of a meta data element based on the ID
Wrap a time average with the n over which it was calculated
Statistics calculated over all documents
Custom exception indicating the user is not authenticated properly (missing / invalid auth token), will be used to return appropriate http error code
Base class for AnnotationGenerators which are updatable, for example ML models, as compared to static models like a statistics based generator
Element displaying an image provided by an URL
Display a meta data element that is an URL of an image
Data class used to store the AnnotateProject used in the annotation process. For example, the selection is not part of this, because it can be very big and is generally unnecessary to store.
Data class representing a user. This model does not contain login information except the unique userIdentifier, and contains an optional userName and the lastAccessTimestamp
DAO for accessing the user collection
View on the User only containing the unique userIdentifier and the userName
Single ValidationError w.r.t. an annotation definition + target
In some contexts, a value might have an associated probability. This is the case when annotations are imported or generated with a probability of how likely this annotation has the value.
Information about a web hook export, containing the export URL, how often it was tries, if it was successful, and possible failure logs for debugging purposes (for example when the web hook returns a 401 unauthorized etc.)