How to Create an Express + TypeORM REST API and Return Data as a Stream
2023-11-20
TypeScriptTypeORMStreamRest-APIExpress
2023-11-20
Recently, I have been learning how to develop REST APIs using Express and how to write tests for them. As part of my learning journey, I have also been exploring TypeScript ORMs capable of handling streams.
A stream is a sequence of data elements made available over time, used for handling large volumes of data or continuous data flow. Streams are useful because they allow for efficient processing of data chunks as soon as they are available, without waiting for the entire data set. This is particularly helpful in managing resources and improving performance in applications like real-time data feeds, file processing, and network communications.
Streams can be particularly useful in various scenarios, such as:
For this purpose, I created the blog-orm repository, which showcases how to interact with a PostgreSQL database using TypeORM and return data as a stream.
For creating the DB Post table you just need to implement a Post.ts entity and data-source.ts:
//entity/Post.ts
import { Entity, PrimaryGeneratedColumn, Column } from "typeorm"
@Entity()
export class Post {
@PrimaryGeneratedColumn()
id: number
@Column()
title: string
@Column()
content: string
@Column()
isPublished: boolean
}
//data-source.ts
import "reflect-metadata"
import { DataSource } from "typeorm"
import { Post } from "./entity/Post"
export const AppDataSource = new DataSource({
type: "postgres",
host: "localhost",
port: 5432,
username: "postgres",
password: "postgres",
database: "blog",
synchronize: true,
logging: false,
entities: [Post],
migrations: [],
subscribers: [],
})
A service could look like that:
//postService.ts
import { AppDataSource } from "./data-source";
import { Post } from "./entity/Post";
export async function getAllStream() {
return await AppDataSource.getRepository(Post).createQueryBuilder("post").stream();
}
And finally the controller:
//controller.ts
import { Request, Response, NextFunction } from "express";
import * as postService from "./postService"
import { pipeline } from "stream/promises";
import Disassembler from 'stream-json/Disassembler';
import Stringer from 'stream-json/Stringer';
export async function getAllStream(req: Request, res: Response, next: NextFunction) {
try {
const stream = await postService.getAllStream();
res.setHeader('Content-Type', 'application/json');
const tokenizer = new Disassembler();
const jsonStream = new Stringer({ makeArray: true });
pipeline(stream, tokenizer, jsonStream, res)
} catch (error) {
console.error(error);
res.status(500).send('An error occurred');
}
}
And that's the whole magic! You can find the complete code in this repository.
I am big football (soccer) fan and follow sports news daily. However, even with this routine, I often find myself unfamiliar with the current standings in the Premier League, Serie A, or other leagues.
In my view, football coverage these days is overwhelmingly occupied with topics like transfer rumors, relegating actual match outcomes to secondary importance.
This observation sparked the idea for my latest project: Sports Dashboard. This Angular-based website displays the standings of the most prominent football leagues across Europe. Additionally, the Austrian Bundesliga features on the site not because it's one of Europe's most critical competitions, but because I reside in Austria and wish to stay informed about local football.
To provide the frontend with data I developed a new REST endpoint: GET https://abelcastro.dev/api/sports/standings/.
The data will be automatically updated every day at 2 AM.
I have implemented a management command import_league_standings.py, which is scheduled to run daily via a cron job on the server.
My goal with this project was to become familiar with Angular, and for me, the best way to achieve that is by working on a real project. In this case, the project turned out to be something truly useful. I hope the Sports Dashboard will also help me to reduce the frequency of my visits to sports news pages.
I wanted to share a little adventure I had while learning Angular. My blog at abelcastro.dev was doing just fine with Django, but I thought, "Why not tinker around with Angular and learn some new skills?" So, I created a playground at ng.abelcastro.dev and decided to see what I could do!
Angular is a fantastic toolbox for making web applications more interactive. Even though my blog was working perfectly with Django, I wanted to learn Angular just for the sake of learning, not necessarily to make it look fancier.
First things first, I set up a separate space for Angular at ng.abelcastro.dev. This way, I could experiment and learn without interfering with the good old Django backend.
To make things flow smoothly, I created a pathway for my Angular app to access the blog's data using a RESTful API at `abelcastro.dev/api/posts´ in my Django backend. This API helps Angular fetch blog posts.
In the Django world, I used the Django REST framework to create the /api/posts endpoint. This way, my Angular app can easily fetch the blog post data it needs.
Inside the Angular app, I used Angular's HttpClient module to fetch data from the Django backend's /api/posts.
You can find the repository for my new angular blog here 🚀.