Document Store

This package provides an document store API for user save data.

Read the primer, installation, and quick start to get started.

A Primer

A primer:

Crayta’s save data is a normal lua table. It might look like this:

local saveData = {
  1 = { _id = 1, name = "DryCoast" },
  2 = { _id = 2, name = "Cereal"}
}

This is conventionally called a key-value store. It’s data that can be retrieved by providing the key. So, if you called self:GetSaveData()[1] on the above save data, you would receive { _id, name = "DryCoast" }.

DocumentStore provides a way to query and update this data, beyond via the key.

For example, let’s say you had this document in the store:

{ _id = 1, name = "Cereal", occupation = "Programmer" }

If I wanted to access this document in save data, I would need to know the key that it’s stored under. Otherwise, I need to iterate through every key in the save data, checking the value for what I want. (In this case, I’m looking for Cereal).

DocumentStore undertakes all of the responsibility for finding the record, so you don’t have to. To find the record above, you would simply call self.db.players:FindOne({ name = "Cereal"})`. This will return the record we want, and we didn’t have to handle loading the save data, iterating it, or anything. We simply ask for the record we want, and we receive it.

The same can be said for updating record. If you want to update a record, you don’t necessarily know where it is. You don’t necessarily know what data the record contains. Let me give an example of a problem I had when developing 2048.

When the user logs into 2048, the first thing that happens is it checks if the save data has been initialized.

function UserScript:Init()
  self.saveData = self:GetSaveData()
  if not self.saveData.initialized then
    self.saveData.highScore = 0
    self.saveData.highestCombo = 0
    self.initialized = true
  end
end

Later on in the code I made a mistake. I called self:SetSaveData({ highScore = value}). After I published the game, I noticed that when I logged in, I lost my highest combo. This happened because calling SetSaveData({ highScore = value }) removed the initialized value from the save data! This is an easy mistake to make, and can be quite demoralizing.

So how does DocumentStore help us in this situation? With the DocumentStore, you’re not concerned with what’s currently in the record, you’re only concerned with the data you want to update. In our highScore example, you would run this:

self.db.scores:UpdateOne({}, { -- Find the first record in the database
  _set = { -- Set a field
    highScore = value -- Set the highScore field to value
  }
})

We can see here that the DocumentStore isn’t concerned with what data is currently stored in the record. It will discretely update just the single field we want updated.

Installation

Installation:

  1. Install the Document Store Package
  2. Drag the Document Store template onto the user
Quick Start

Quick Start

DocumentStore will manage many databases for you, but by default there is a single document store called default. If you want to add more (for example, maybe a stats database, or an inventory database) simply add another documentStoreScript to the documentStore script folder on the user, and set the id property to the name of the database.

All of the interactions with the database take place through the documentStoresScript. A common setup on a user script looks like this:

function MyScript:Init()
  self.db = self:GetEntity().documentStoresScript
end

With this setup, you can access any database setup via self.db.<database>. For example, the default database would be self.db.default.

The document store API provides basic CRUD functionality. It aims to replicate the MongoDB API, you can read more about the MongoDB API here: MongoDB CRUD Operations | MongoDB

The supported operations and selectors are listed at the end of this forum post, and will be updated as I improve support.

  • InsertOne(record)
  • InsertMany({ …records })
  • UpdateOne(query, updateOperation)
  • UpdateMany(query, updateOperations)
  • ReplaceOne(query, newRecord)
  • Find(query, options)
  • FindOne(query, options)
  • DeleteOne(query)
  • DeleteMany(query)

There is also a utility function WaitForData that you can call inside of a schedule to pause execution until the save data is loaded from Crayta.

Basic usage is simple

To find a record, run self.db.default:FindOne({ name = "Cereal" })
To insert a record, run self.db.default:InsertOne({ name = "Cereal" }).
To update a record, run self.db.default:UpdateOne({ name = "Cereal" }, { _set = { name = "notCereal" } })
To delete a record, run self.db.default:DeleteOne({ name = "Cereal" })
To replace a record, run self.db.default:ReplaceOne({ name = "Cereal" }, { name = "notCereal" })

All of the CRUD methods return record(s). All records have an _id property. If you query on this property, the find will be O(1). Using any other property will be O(n).

Advanced

InsertOne

InsertOne inserts a single record into the database.

function TestScript:TestInsertOne()
	local result = self.db:InsertOne({ name = "Foo", age = 18 })
end

--[[
Returns
{
  _id = "54ad197f-ccee-46c4-91b1-194a75b675e2",
  name = "Foo",
  age = 18,
}
]]--

InsertMany

Inserts many records into the database.

function TestScript:TestInsertMany()
  local records = self.db:InsertMany({
    { name = "Foo", age = 18 },
    { name = "Bar", age = 24 },
  })
end

--[[
Returns
{{
  _id = "54ad197f-ccee-46c4-91b1-194a75b675e2",
  name = "Foo",
  age = 18
},
{
  _id = "4315ad82-eb40-4add-91d8-92c080ff4ead",
  name = "Bar",
  age = 24
}}
]]--

UpdateOne(filter, operations)

Updates a single record in the database, using filter as the query. operations is a table of supported operations, as listed on at the end of this post. Operations should behave similarly to those listed here: Update Operators — MongoDB Manual

This function will update the first record matching the query.

function TestScript:TestSetOperation()
	
	self.db:InsertOne({ name = "Foo", age = 18 })
	
	local record = self.db:UpdateOne({ name = "Foo" }, { _set = { name = "Bob" } })
end

--[[
{
  _id = "4315ad82-eb40-4add-91d8-92c080ff4ead",
  name = "Bob"
}
]]--

UpdateMany(filter, operations)

Works similarly to UpdateOne, however it updates all records that match the query.

function TestScript:TestSetOperation()
	
	self.db:InsertMany({ { name = "Foo", age = 18 },  { name = "Foo", age = 24 } })
	
	local record = self.db:UpdateMany({ name = "Foo" }, {  _set = { name = "Bob" } })
end

--[[
Returns
{{
  _id = "54ad197f-ccee-46c4-91b1-194a75b675e2",
  name = "Bob",
  age = 18
},
{
  _id = "4315ad82-eb40-4add-91d8-92c080ff4ead",
  name = "Bob",
  age = 24
}}
]]--

ReplaceOne(filter, record)

Instead of updating a record, replace the first matching record with the provided document.

function TestScript:TestReplaceOne()
        self.db:InsertOne({ name = "Foo", age = 18 })
	local record = self.db:ReplaceOne({ name = "Foo"  },  { name = "Dolittle", age = 101 })
end

--[[
{
  _id = "54ad197f-ccee-46c4-91b1-194a75b675e2",
  name = "Dolittle",
  age = 101
}
]]--

Find(query)

Finds all matching records for a query.

function TestScript:TestFind()
	local inserted = self.db:InsertMany({
		{ name = "Foo", age = 99 },
		{ name = "Foo", age = 36 }
	})
	
	local records = self.db:Find({ name = "Foo" })
end

--[[
{
{{
  _id = "54ad197f-ccee-46c4-91b1-194a75b675e2",
  name = "Foo",
  age = 99
},
{
  _id = "4315ad82-eb40-4add-91d8-92c080ff4ead",
  name = "Foo",
  age = 36
}}
}
--]]

FindOne(query)

Matches the first matching record for a given query

function TestScript:TestFindOne()
	local inserted = self.db:InsertMany({
		{ name = "Foo", age = 99 },
		{ name = "Foo", age = 36 }
	})
	
	local record = self.db:FindOne({ name = "Foo" })
end

--[[
{
  _id = "4315ad82-eb40-4add-91d8-92c080ff4ead",
  name = "Foo",
  age = 99
}
]]--

DeleteOne(query)

Deletes the first matching record out of the database

function TestScript:TestDeleteOne()
	self.db:InsertOne({ name = "Foo" })
	
	self.db:DeleteOne({ name = "Foo" })
	
	return assertEqual(0, #self.db:Find())
end

DeleteMany(query)

Deletes all matching records for a query. If the query is omitted, deletes all records.

function TestScript:TestDeleteMany()
	self.db:InsertMany({ { name = "Foo" }, { name = "Foo" }, { name = "Bar" } })
	
	self.db:DeleteMany({ name = "Foo" })
	
	return assertEqual(1, #self.db:Find())
end

Supported operations for update:

  • _set - set a field on the matched record(s)
  • _inc - increment a field by the given value
  • _min - only set the field if the given value is smaller than the field value
  • _max - only set the field if the given value is larger than the field value
  • _unset - remove the value from a field
  • _rename - rename a field
  • _mul - multiply a field by a given value
  • _setOnInsert - set a field, only if the record is newly inserted
  • _addToSet - Add a value to an array, if the value doesn’t exist already
  • _pop - Remove the last value of an array
  • _pull - Removed all elements matching the given query
  • _push - Add elements to an array
  • _pullAll - removes all elements from an array matching the given values

Some quick examples of these operators

self.db:UpdateOne({ name = "Foo" }, { _set = { name = "Bar" } })
self.db:UpdateOne({ value = 1 }, { _inc = { value = 2 } }) -- value becomes 3
self.db:UpdateOne({ value = 1 }, { _max = { value: 3 } }) -- value becomes 3
self.db:UpdateOne({ value = 1}, { _rename = { value = "foo" } }) -- { foo = 1 }

Supported selectors for find

  • _eq - match a field equal to the given value
  • _gt - match a field greater than a given value
  • _lt - match a field less than a given value
  • _lte - match a field less than or equal to a given value
  • _gte - match a field greater than or equal to a given value
  • _and - match a record if all expressions in an array resolve true
  • _ne - match a record if a field is not equal to the given value
  • _nin - match a record if the field is not in the given array of values
  • _in - match a record if the field is in an array of given values
  • _size - Match a record whose given array is the given size
  • _elemMatch - Match a record whose given array contains an element that matches all given selectors
  • _all - Match a record whose given array contains all values
  • _mod - Match a field that returns the given remainder when divided by the given divisor
    Some examples of these selectors:
  • _type - Match a field that matches the given type
  • _exists - Match a field that exists or not
  • _not - Matches a field that doesn’t match the given queries
self.db:FindOne({ value = { _gt = 2 } })
self.db:FindOne({
  value = {
    _in = { 1, 2, 3, 4, 5 }
  }
})
self.db:FindOne({ value = { _eq = 3 } })
self.db:FindOne({
  value = {
    _and = {
      { value = { _gt = 10 } },
      { value = { _nin = { 20, 30, 40, 50, 60, 70 } } }
    }
  }
})
6 Likes

April 20th update

  • Added upsert option to UpdateOne and UpdateMany
  • Can now query using dot notation
function TestScript:TestFindOneNested()
    self.db:InsertOne({
        author = {
            name = "Bob"
        }
    })
    
    local query = {}
    query["author.name"] = "Bob"
    
    local record = self.db:FindOne(query)
    
    return assertEqual("Bob", record.author.name)
end
  • Added _setOnInsert operation
  • Added _ positional array operation
  • Added _[] positional array operation

April 20th Update (contd.)

  • Added support for the _[<identifier>] operator
  • added support for the arrayFilter option on UpdateMany and UpdateOne methods

-- Set all values in the values property greater than 3, to 6
function TestScript:TestFilterArray()
	self.db:InsertOne({ values = { 1, 2, 3, 4, 5 } })

	local query = {
		_set = {}
	} 
	query._set["values._[test]"] = 6
	
	local record = self.db:UpdateOne({}, query, { 
		arrayFilters = { { test = {
			_gt = 3
		}} }
	})
	
	return assertEqual(6, record.values[4])
end
  • Added support for the _addToSet operator
  • Added support for the _pop operator
  • Added support for the _pull operator
2 Likes

April 22nd update

  • Added a documentStoresScript
  • Added an id property to the documentStoreScript

You can now use the UseDb(name) function on documentStoresScript to segment your stores into logical tables.

For example, if you wanted to have a database for items and stats on the user, you can set up your user template like so:

To switch between the databases, or decide which one to use, you would use documentStoresScript.

function MyScript:Init()
  self.db = self:GetEntity().documentStoresScript

  self.db.items:InsertOne({ name = "Sword" })
  self.db.stats:InsertOne({ name = "Strength" })
end

April 23rd update

  • Added the _not query operator
  • Fixed the _and query operator - the syntax was wrong. It now matches the mongo syntax
  • Added the _nor query operator
  • Added the _exists query operator
  • Added the _type query operator
  • Added the _mod query operator
  • Added the _all query operator
  • Added the _elemMatch query operator
  • Added the _size query operator
  • Added the _push update operator
  • Added the _pullAll update operator
  • Added the _each update modifier
  • Added the _position update modifier
  • Added the _slice update modifier
  • Added the _sort update modifier
  • Fixed a bug preventing you from setting a property as an empty table
  • DeleteOne method now returns the record deleted (edited)
1 Like

This is such a useful package. I’m going to see how I can incorporate it in future. Thank you!

  • Update/Insert/Delete methods now broadcast an OnRecordUpdated/Inserted/Deleted event to world, with the parameters (table, record)
  • Fixed an issue where upsert just straight up didn’t do anything
  • Fixed an issue where _inc on a nil value would throw an error instead of incrementing as if it was 0
  • Upserting a document with an _id query will now use the id from the query when inserting
1 Like
  • The second parameters to the find methods is now options, not projections. You can still pass projections in the project option
  • Added skip, sort, and limit options to the find methods
  • Fixed a bug where upserting a new record on the client would only cause an insert on the server until you left and re-entered the game
  • Document Stores now have a version, allowing you to create a new dataset on a per-store basis.
  • Fixed a bug where options were not being propagated to the query during a Find operation
  • Fixed a bug where inserting a record on the local client would lose the record id