Developing AcadeMix Backend

Gravewalker · May 23, 2020, 2:47pm

We need to come up with a solid structure for the backend of the AcadeMix project.

As we have discussed in the weekly meeting first we are gonna come up with the structure, identify the database requirement. After that, we will identify the necessary endpoints and post it here so that we can proceed with the UI development simultaneously.

YohanAvishke · May 23, 2020, 8:33pm

Hi all,

Following are my suggestions for the database and API design.

Database

We have decided to implement 3 main entities named category, sub_category, and item I have included a design idea based around the above entities to implement a database.

Adding relational tables for each entity to map relations with a language entity.

Above I have created a new table for the three main entities(*_translation). These tables will help to match language types with the data. Even though this approach is a bit complicated and requires a few more tables, I think this is better as if someday this project expands to an international level we can add support for more languages easily without breaking the code.
Basically we get the ability to manage languages in the database.

API

For the first phase following are the APIs we are planning to develop.

/languages
- GET - all languages
  Payload - [ { id:1, locale:"un-en" }, ... ]
/categories
- GET - all categories
  Payload - [ { id:1, name:"Teacher" }, ... ]
- POST - a category
/sub-categories
- GET - sub-categories by category
  Payload - [ { id:1, name:"Courses on Machine Learning" } , ... ]
- POST - a category
/items
- GET - item by id
  Payload - { id:1, name:"beginners guide to machine learning", description:"Lorem ipsum ...", link:"LINK_TO_THE_COURSE" }
- GET - items by sub-category
  Payload - [ { id:1, name:"beginners guide to machine learning", description:"Lorem ipsum ...", link:"LINK_TO_THE_COURSE" }, ... ]
- POST - an item

These are just a few suggestions, your reviews will help a lot to reach a final conclusion. So feel free to comment .

Thank you

akshika47 · May 23, 2020, 10:10pm

This is great progress @YohanAvishke! Love the attention to detail. One quick comment looking at the ER diagram. You might need another table to map items which falls under multiple categories. I would also have a variable in item table to have a description which can be used from us to add any additional information related to the content. What do you think?

YohanAvishke · May 24, 2020, 5:09am

Hi @akshika47 thanks a-lot for the response

Good point @akshika47, in this case, it’s better to change relationships of entitiees as below

Now it’s possible to,

Duplicate sub-categories among categories.
Duplicate items among sub-categories.

Which automatically results in items to be duplicated among categories.

This is already included in the item_translation entity. As you can see below, we are persisting both name and description.
By the end of the day, each item will have an id, name, description, and a link

A new normalized diagram with the above changes is attached below, Please note that I have done some variable type changes too.

jaye · May 24, 2020, 7:16am

This is awesome @YohanAvishke!

Just a suggestion; Since we know we know this is gonna get translated only for 3 languages, can’t we add the translations inside the entity?

ex: category( id , name_en, name_si, name_ta )
@akshika47 Are there any use cases in which a subcategory has multiple categories?

akshika47 · May 24, 2020, 10:01am

There is no use case for subcategory having multiple sub categories but there is a use case for an item being under more than 1 category or sub category. that needs to be represented.
for example “Khan Academy” could be under students section and teachers section. Similarly it could fall under maths as well as some other sub category.

cc : @YohanAvishke

YohanAvishke · May 24, 2020, 10:22am

Thanks a-lot @jaye

I think this is a bad designing pattern due to the following reasons,

Tables will depend on language types.
By creating a new table for translation we can pretty much have an independent table structure. Expanding the database further can be done without having to worry about changing already implemented tables.
In conclusion we will have a more solid table structure.
We will have to worry about translating every bit of data(names and descriptions) if we follow the above design. Otherwise we will have to leave unnecessary null columns.
But by having a new table for translations, we can do implementations on top of our APIs without worrying about multi-language support. Which can be added later, so the development process will be quicker.
Also I think in a nER scenario it makes more sense to create new relationships whenever possible instead of bulking all the data into a single table.

YohanAvishke · May 25, 2020, 2:05am

Hi all, after 2020-05-23T18:30:00Z's call. I have added the changes we discussed to the ERD.

jaye · May 25, 2020, 3:00am

@shyamal @DannyB Do you have any suggestions?

shyamal · May 25, 2020, 3:29am

Hi @jaye and team

This is looking neat assuming below.

Every item will follow the same hierarchy. Category, Sub Category, Item.

General comment, if you are uncertain about the possibility of adding more columns later with unpredictable relationship better to use NoSQL. But this seems like a relatively predictable requirement thus using SQL would be a good idea.

Gravewalker · May 28, 2020, 3:46am

2nd Dev Meeting on Developing Backend

Held on: 2020-05-27T15:30:00Z
Attendees: @jaye @YohanAvishke @Gravewalker

Meeting minutes:

Reviewed the WIP PR on data retrieval endpoints.
Had a discussion on the structure of the payload.

Conclusion
After a discussion on how the payload should structure and the best possible way to extract it in that form from the database, we came to a conclusion that we should continue using only Spring Jpa to extract data and should avoid using native or JPQL custom queries unless the particular query can’t be done with spring JPA repository. After extracting the data, to avoid JSON payload from going into a recursion we decided to use JSON ignore.

Here’s a sample payload of the category

[
    {
        id:1,
        translations: [
            {
                "id":1
                "language":{
                    id:1,
                    locale:"en-un"
                },
                name:"category1"
            },
            {
                "id":2
                "language":{
                    id:2,
                    locale:"si"
                },
                name:"කාණඩ 1"
            }
        ]
    }
]

jaye · May 28, 2020, 4:31am

Thanks for the update @Gravewalker,

Btw, I think It’s better if we could generate the payload like this:

[
    {
        id:1,
        translations: {
              "en-us" : {
                  "name": "Category 1"
               },
              "si-lk" : {
                  "name": "කාණ්ඩය 1"
               },
        }
    }
]

No need to include this for the initial PR. But it’s better to have a payload like this.
cc: @YohanAvishke

Thanks.

Gravewalker · May 29, 2020, 1:57am

After the code review of the still open PR, we decided to move on from the current folder structure to a more project-oriented structure.

Current structure.

src/
└── main
    ├── java
    │   └── org
    │       └── sefglobal
    │           └── core
    │               ├── AdminApplication.java
    │               ├── config
    │               │   └── SecurityConfig.java
    │               ├── controller
    │               │   ├── AcademixController.java
    │               │   ├── CertificateController.java
    │               │   ├── MultiverseController.java
    │               │   ├── MultiverseLinkController.java
    │               │   └── UserController.java
    │               ├── exception
    │               │   ├── APIException.java
    │               │   ├── BadRequestException.java
    │               │   ├── ResourceNotFoundException.java
    │               │   └── UnauthorisedException.java
    │               ├── model
    │               │   ├── Ambassador.java
    │               │   ├── AuditModel.java
    │               │   ├── Category.java
    │               │   ├── CategoryTranslation.java
    │               │   ├── Certificate.java
    │               │   ├── Engagement.java
    │               │   ├── Event.java
    │               │   ├── identity
    │               │   │   └── EngagementIdentity.java
    │               │   ├── Item.java
    │               │   ├── ItemTranslation.java
    │               │   ├── Language.java
    │               │   ├── Link.java
    │               │   ├── SubCategory.java
    │               │   ├── SubCategoryTranslation.java
    │               │   ├── University.java
    │               │   └── User.java
    │               ├── projections
    │               │   ├── CustomCategory.java
    │               │   ├── CustomItem.java
    │               │   ├── CustomLanguage.java
    │               │   ├── CustomSubCategory.java
    │               │   └── CustomTranslation.java
    │               ├── repository
    │               │   ├── AmbassadorRepository.java
    │               │   ├── CategoryRepository.java
    │               │   ├── CertificateRepository.java
    │               │   ├── EngagementRepository.java
    │               │   ├── EventRepository.java
    │               │   ├── ItemRepository.java
    │               │   ├── LanguageRepository.java
    │               │   ├── SubCategoryRepository.java
    │               │   ├── UniversityRepository.java
    │               │   └── UserRepository.java
    │               ├── service
    │               │   ├── AcademixService.java
    │               │   ├── AuthenticationService.java
    │               │   ├── CertificateService.java
    │               │   ├── DataLoader.java
    │               │   └── LanguageService.java
    │               └── util
    │                   └── Status.java
    └── resources
        ├── application.properties
        └── application.properties.example

Suggested changes: Create a package under org.sefglobal.core called academix and add necessary files.

shyamal · May 30, 2020, 1:22pm

Hi Team,
If you don’t mind research little bit about “Clean Architecture”. Provide very flexible extensive approach to organize your code that may grow extensively yet can accommodate changes with less headache.

Key. Only benefits If code base grows eventually with different perspectives such as domain or features, technology changes like db or api etc.

On a different note. How do you connect on the 7pm call?

Gravewalker · May 30, 2020, 1:48pm

@shyamal
You can use this meet link to connect https://meet.google.com/chs-konp-dst

jaye · June 1, 2020, 3:25am

@Gravewalker @YohanAvishke
Any updates from the data retrieval API?

YohanAvishke · June 1, 2020, 3:56am

@Gravewalker is still working on it. @Gravewalker can you please give an update. State if you’re facing any blockers.

Gravewalker · June 1, 2020, 7:09am

I updated the PR with the requested amendments.
If no further amendments are needed data retrieval part of the API is completely implemented with this PR.

YohanAvishke · June 1, 2020, 7:59am

@Gravewalker there are still a few unresolved discussion which seems to be outdated. Can you please add comments to them pointing to the commit that fixed them.

Gimhan_minion · June 2, 2020, 3:34pm

@YohanAvishke Aiya Where are we up to now on “Creating API’s”? Do you need any help from my end?

Topic		Replies	Views
IRC Page Improvements Site Feedback	4	285	November 12, 2020
Design the Initial ERD of ScholarX ScholarX	31	2251	October 22, 2020
Design Forum: Updating AcadeMix admin panel Development	10	747	July 12, 2020
Updating AcadeMix page Development	157	3250	July 27, 2020
Updating admin panel to control academix backend operations Development	9	509	June 16, 2020

Developing AcadeMix Backend

Related topics