Global entities size



  • What is the maximum data size of a global entity on BrainCloud?

    I assume that is related to the maximum MongoDB BSON document size (16 megabytes). Is this true?


  • brainCloud

    Hi,

    Yes and no.

    You are correct that we store all our entity types, Global, User, Group and Custom Entities in MongoDB, and thus their 16 MB limit is the overall technical limit.

    That said, what we recommend is considerably smaller.

    In general, we recommend that apps keep entities to less than 1MB - and for entities that will be updated often, the sweet spot is generally no larger than 50-200Kb. (Mobile devices especially can have challenges uploading more than 200Kb in an update).

    Global Entities are often mostly read-only though - and thus those sizes can get larger if they have to. Using Global Entities to store read-only level data that's 1-2MB in size will probably be fine.

    Some other guidelines regarding Global Entities:

    • There is limited indexing of global entities. In general, performance of global entities will slow down after 1000 or so entities - especially if you are retrieving entities using anything other than it's entityIndexedId, entityId or entityType. (Note - the new Custom Entities fixes this limitation)
    • For this reason, we do not recommend writing a Global Entity for each user in the system (User Entities and/or Owned Custom Entities are much better for that)
    • Updates to Global Entities should be limited. You don't want a situation where thousands of users are attempting to concurrently update the same Global Entity.
      • The versioning system will necessitate tons of retries (which will get worse as the population of your app gets larger), or you'll lose updates if you bypass the versioning.
      • This problem is especially compounded if your entities are large. The IncrementGlobalEntityData() call is a partial improvement on this (though not perfect).
      • Consider using global stats instead - or ways to break up the data so that fewer users will be trying to update a single entity at the same time.

    Some additional entity guidelines - these apply to other entity types as well:

    • The smaller your entities are, the faster they will be to read and write.
    • The more entities you have (of a type), the slower it is to find the appropriate entities to read or write. This is why you should generally keep global entities to hundreds or less - and user entities to ~dozens (per user). Note - the custom indexes of Custom Entities highly alleviate this concern.
    • There is therefore a give-and-take between fewer large entities and many small entities. Within reason, more smaller entities is generally better (but without surpassing the guidelines of "hundreds of global entities" or "dozens of user entities (per user)".

    Pro-tip: Consider the access patterns of the data when designing your entities. i.e. Consider segregating the data based upon when and how often the data is updated.

    If you have game with exploration, you might want to regularly update a player's location, direction, health stats, current weapon, etc. Give that data it's own entity, and update it every 20 seconds +/- - whatever makes sense for your game. The data will be small - likely less than 5Kb.

    In particular, keep that data separate from the structure that describes the player's full meta-data (their name, classs, specialization attributes, overall inventory, spells learned, etc - things that rarely change) - that data could easily be hundreds of Kb.

    And definitley do consider Custom Entities - they're the super-charged versions of Global and User entities. We'd expecially recommend them instead of Global Entities, especially for cases where you will have large datasets with many ways in which that data will be accessed.

    Hope that helps!

    Paul.


  • brainCloud

    Hi,

    Yes and no.

    You are correct that we store all our entity types, Global, User, Group and Custom Entities in MongoDB, and thus their 16 MB limit is the overall technical limit.

    That said, what we recommend is considerably smaller.

    In general, we recommend that apps keep entities to less than 1MB - and for entities that will be updated often, the sweet spot is generally no larger than 50-200Kb. (Mobile devices especially can have challenges uploading more than 200Kb in an update).

    Global Entities are often mostly read-only though - and thus those sizes can get larger if they have to. Using Global Entities to store read-only level data that's 1-2MB in size will probably be fine.

    Some other guidelines regarding Global Entities:

    • There is limited indexing of global entities. In general, performance of global entities will slow down after 1000 or so entities - especially if you are retrieving entities using anything other than it's entityIndexedId, entityId or entityType. (Note - the new Custom Entities fixes this limitation)
    • For this reason, we do not recommend writing a Global Entity for each user in the system (User Entities and/or Owned Custom Entities are much better for that)
    • Updates to Global Entities should be limited. You don't want a situation where thousands of users are attempting to concurrently update the same Global Entity.
      • The versioning system will necessitate tons of retries (which will get worse as the population of your app gets larger), or you'll lose updates if you bypass the versioning.
      • This problem is especially compounded if your entities are large. The IncrementGlobalEntityData() call is a partial improvement on this (though not perfect).
      • Consider using global stats instead - or ways to break up the data so that fewer users will be trying to update a single entity at the same time.

    Some additional entity guidelines - these apply to other entity types as well:

    • The smaller your entities are, the faster they will be to read and write.
    • The more entities you have (of a type), the slower it is to find the appropriate entities to read or write. This is why you should generally keep global entities to hundreds or less - and user entities to ~dozens (per user). Note - the custom indexes of Custom Entities highly alleviate this concern.
    • There is therefore a give-and-take between fewer large entities and many small entities. Within reason, more smaller entities is generally better (but without surpassing the guidelines of "hundreds of global entities" or "dozens of user entities (per user)".

    Pro-tip: Consider the access patterns of the data when designing your entities. i.e. Consider segregating the data based upon when and how often the data is updated.

    If you have game with exploration, you might want to regularly update a player's location, direction, health stats, current weapon, etc. Give that data it's own entity, and update it every 20 seconds +/- - whatever makes sense for your game. The data will be small - likely less than 5Kb.

    In particular, keep that data separate from the structure that describes the player's full meta-data (their name, classs, specialization attributes, overall inventory, spells learned, etc - things that rarely change) - that data could easily be hundreds of Kb.

    And definitley do consider Custom Entities - they're the super-charged versions of Global and User entities. We'd expecially recommend them instead of Global Entities, especially for cases where you will have large datasets with many ways in which that data will be accessed.

    Hope that helps!

    Paul.



  • Thank you for your detailed answer!


  • brainCloud

    No problem - happy to help!

    Paul.