Folks from MongoDB have documented a very nice approach of using a data model for tree-like structures: they call it *Materialized Paths *. Consider following: If I populate this hierarchy using following code: db.categories.insert( { _id: "Books", path: null } ) db.categories.insert( { _id: "Programming", path: ",Books," } ) db.categories.insert( { _id: "Databases", path: ",Books,Programming," } ) db.categories.insert( { _id: "Languages", path: ",Books,Programming," } ) db.categories.insert( { _id: "MongoDB", path: ",Books,Programming,Databases," } ) db.categories.insert( { _id: "dbm", path: ",Books,Programming,Databases," } ) I can easily retrieve, for example, all descendants of *Programming*: db.categories.find( { path: /,Programming,/ } ) Now I'm considering of moving this database into Google Cloud Datastore , but I'm concerned about how painful would be to perform queries for such filtering? Can someone provide an example? If it's relevant, backend is written in Go.

I have a very simple requirement. I have an application where users log in, add/remove/view friends, and can send/receive (also view the message you sent and received) a message (message is more like an email, not a real time chat) from a friend. When they receive the message, the application needs to mark the message as 'read'. The messages must also come ordered by newest to oldest. My thoughts: KINDS: USER username (ID) password friends (will contain a max of 100 friends). --------------------- MESSAGE msg_id (ID) from (indexed) to (indexed) timestamp (indexed) msg hasRead 1) User would login using their username and password 2) User could get friends by getting their USER entity based on their username 3) User could add/remove friends by getting the entity of user1 and user2 and either adding or removing friend via transaction to make sure they are consistent. 4) User could get all the message they have sent by using indexing the the 'from' attribute (limit of 10 message per request). The same could be done to view all the messages they have received by using the 'to' attribute. When the message has been seen for the first time I would go to the message (1 get) and update the entity (1 write + (4 writes x 3) = 13 writes to update the entity). My major concern - If a user gets 10 messages, this will require 10 get requests plus if all 10 messages are new I will need to update each entity and if all of them are new that is (10 x 14) 140 writes. Then if I get another 10 message for this user the same process and this could all add up very quickly. I then thought of creating an entity to store all the sent/received messages in a string for a user inside of a single entity: MESSAGE user_entitynum1 (ID) messages This way I could store all the message (under 1mb) inside of this one entity, but I would have to keep track of which entity each user is at (if the user exceeds the 1mb limit I will have to create a new entity). This proved to also not be too efficient because if I send a message to perform 2 gets to see which message entity I am currently at and which message entity they are currently at. From there I must now use another 2 reads to get those entities and another 4 writes to update them both (considering I do not need to create another entity if it is full). I was looking for any ideas to accomplish what I need in a more efficient way. I have nothing implemented yet so I am open to ANY ideas. My main concern is for this to be efficient, Cheers!

I have two kinds: USER and MESSAGES. A user can send/receive many messages to/from another user. Thinking in a relational way the relationship would suggest a user can have many messages, and a message can only be created by one user. So if user A sends a message to User B, both user A should be able to see the message they sent, and user B should be able to see the message they received. I was thinking first of storing all sent/received messages inside of each user object, but since there is a 1mb limit for entities this is not an option as some users can retrieve many more messages than this. Second I was thinking of creating two kinds, USER and MESSAGES.The messages kind would have a to and from property both which would be indexed so that I can get all the messages a user has sent and received. As a result each message would be its own entity and this is where it is problematic. Lets say a User is deleted, I know have to delete all the message sent and received by this user. If a user has sent though sands of messages, the cost of removing each message will be very expensive. I was wondering what would be an efficient way to model this. I am be to Google Datatore and would be open to any ideas as I have nothing implemented yet.

I am trying to learn about Google Datastore and creating a basic application to have some practical experience with it opposed to just reading the docs. I would like to create a basic social application where users can send messages to their friends and view the messages their friends have sent them. My Basic App Requirements: **Users Section:** - A user registers to the app. The user can then log in by using their username/email and checking if the password is correct and the status must be VALID and not something like DEACTIVATED. - A user can also be deleted. When a user is deleted If a user deletes their account their references from all their friends need to be removed and messages needs to be removed (or hidden). This is obviously expensive. *Users Model* -username (need to be able to make equality searches on this field. e.g select * from kind where username = 'Bob') (index) -email (need to be able to make equality searches on this field. e.g select * from kind where email = 'Bob@gmail.com') (index) -fullname -password -status (need to be able to make equality searches on this field. e.g select * from kind where status = 'DEACTIVATED') (index) -time_created **Friends Section:** - A user can query to find all of their friends (Accepted and Pending). - A user can add another user to be friends (make sure relationship doesn't already exist and make sure user exists). - A user can accept/reject a friend request. - A user may have a max of 100 friends. *Friends Model* -User1 (index) -User2 (index) -status -time_created **Messages Section:** - A user may see all the messages they sent/received to/from confirmed friends - A user can send a message to a confirmed friend. - A user may receive a message to a confirmed friend. - Messages will be paginated in the application so 10 messages at a time come up sorted by the newest coming first. *Messages Model* -from (need to be able to make equality searches on this field. e.g select * from kind where from = 'Bob') -to (need to be able to make equality searches on this field. e.g select * from kind where to = 'Bob') -msg -time_created (need to sort on this so that the newest messages come first) My plan is to make 3 kinds: Users, Friends, and Messages (each would have the attributes shown above). Some concerns I have is regarding atomic operations. For users, would it be wiser to create a one way relationship between friends or two-way. If two-way, say the friends request gets accepted or rejected, would the application be atomic? Another question is if a user gets removed, how best to handle this to be inexpensive and keep data consistent. Is this a good model for this basic/popular type of social application? I am very interested to hear of anyone suggestion how they would implement this basic application and the reasons why. Cheers!

I am working on a personal project where I am creating a small CMS using PHP as the developing language, but I want to be able to deploy it to the Google App Engine and I want to use its Google Datastore (NoSQL). The question I have is how to do relationship between kinds? **For example:** - Kind 1: call category - Kind 2: call tags - Kind 3: call post I want the posts to be able to have one category, but multiple tags. I am having trouble visualizing it. I am coming from SQL world so having the ID of the category and the IDs of the tags to them JOIN them will take care of the same issue. **Note:** if there is some tutorial or reading that you could share I will appreciate it.

I have developed an application in Java that is registered on Google App Engine. I need to store some data on data store. But, I am completely unaware about its implementation. Actually, I have implemented OAuth, so I want to store data of particular user who has logged in. Is it possible to separate out data stores for different users ? Also, there is some quota limitation. How much data can we store for whole application ?

How to migrate data from Google AppEngine's Datastore to other database (not Datastore)? There is appcfg.py download_data but it produces sqlite3 file with empty bulkloader_database_signature and useless result tables.