Browse Source

Merge pull request #8494 from salim-b/litestream-config

feat: improve Litestream config
pull/8505/head
Mert E 1 month ago committed by GitHub
parent
commit
eacfa2ab3d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
  1. 133
      packages/noco-docs/docs/020.getting-started/050.self-hosted/020.environment-variables.md
  2. 11
      packages/noco-docs/docs/070.fields/040.field-types/050.custom-types/010.attachment.md
  3. 45
      packages/nocodb/Dockerfile
  4. 26
      packages/nocodb/Dockerfile.local
  5. 22
      packages/nocodb/docker/litestream.yml
  6. 35
      packages/nocodb/docker/start-litestream.sh
  7. 2
      packages/nocodb/docker/start-local.sh
  8. 96
      packages/nocodb/litestream/Dockerfile

133
packages/noco-docs/docs/020.getting-started/050.self-hosted/020.environment-variables.md

@ -6,68 +6,75 @@ tags: ['Open Source']
keywords : ['NocoDB environment variables', 'NocoDB env variables', 'NocoDB envs', 'NocoDB env']
---
For production use-cases, it is **recommended** to configure
- `NC_DB`,
- `NC_AUTH_JWT_SECRET`,
- `NC_PUBLIC_URL`,
For production use cases, it is **recommended** to set at least:
- `NC_DB`
- `NC_AUTH_JWT_SECRET`
- `NC_PUBLIC_URL`
- `NC_REDIS_URL`
| Variable | Comments | If absent |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------|
| NC_DB | See our example database URLs [here](https://github.com/nocodb/nocodb#docker). | A local SQLite will be created in root folder if `NC_DB` is not provided |
| NC_DB_JSON | Can be used instead of `NC_DB` and value should be valid knex connection JSON | |
| NC_DB_JSON_FILE | Can be used instead of `NC_DB` and value should be a valid path to knex connection JSON | |
| DATABASE_URL | Can be used instead of `NC_DB` and value should be in JDBC URL format | |
| DATABASE_URL_FILE | Can be used instead of `DATABASE_URL` and value should be a valid path to file containing JDBC URL format. | |
| NC_AUTH_JWT_SECRET | JWT secret used for auth and storing other secrets | A random secret will be generated |
| PORT | For setting app running port | `8080` |
| DB_QUERY_LIMIT_DEFAULT | Pagination limit | 25 |
| DB_QUERY_LIMIT_GROUP_BY_GROUP | Group per page limit | 10 |
| DB_QUERY_LIMIT_GROUP_BY_RECORD | Record per group limit | 10 |
| DB_QUERY_LIMIT_MAX | Maximum allowed pagination limit | 1000 |
| DB_QUERY_LIMIT_MIN | Minimum allowed pagination limit | 1 |
| NC_TOOL_DIR | App directory to keep metadata and app related files | Defaults to current working directory. In docker maps to `/usr/app/data/` for mounting volume. |
| NC_PUBLIC_URL | Used for sending Email invitations | Best guess from http request params |
| NC_JWT_EXPIRES_IN | JWT token expiry time | `10h` |
| NC_CONNECT_TO_EXTERNAL_DB_DISABLED | Disable Project creation with external database | |
| NC_INVITE_ONLY_SIGNUP | Removed since version 0.99.0 and now it's recommended to use [super admin settings menu](/account-settings/oss-specific-details#enable--disable-signup). Allow users to signup only via invite URL, value should be any non-empty string. | |
| NUXT_PUBLIC_NC_BACKEND_URL | Custom Backend URL | ``http://localhost:8080`` will be used |
| NC_REQUEST_BODY_SIZE | Request body size [limit](https://expressjs.com/en/resources/middleware/body-parser.html#limit) | `1048576` |
| NC_EXPORT_MAX_TIMEOUT | After NC_EXPORT_MAX_TIMEOUT, CSV gets downloaded in batches | Default value 5000(in millisecond) will be used |
| NC_DISABLE_TELE | Disable telemetry | |
| NC_DASHBOARD_URL | Custom dashboard URL path | `/dashboard` |
| NC_GOOGLE_CLIENT_ID | Google client ID to enable Google authentication | |
| NC_GOOGLE_CLIENT_SECRET | Google client secret to enable Google authentication | |
| NC_MIGRATIONS_DISABLED | Disable NocoDB migration | |
| NC_MIN | If set to any non-empty string the default splash screen(initial welcome animation) and matrix screensaver will disable | |
| NC_SENTRY_DSN | For Sentry monitoring | |
| NC_REDIS_URL | Custom Redis URL. Example: `redis://:authpassword@127.0.0.1:6380/4` | Meta data will be stored in memory |
| NC_DISABLE_ERR_REPORT | Disable error reporting | |
| NC_DISABLE_CACHE | To be used only while debugging. On setting this to `true` - meta data be fetched from db instead of redis/cache. | `false` |
| AWS_ACCESS_KEY_ID | For Litestream - S3 access key id | If Litestream is configured and `NC_DB` is not present. SQLite gets backed up to S3 |
| AWS_SECRET_ACCESS_KEY | For Litestream - S3 secret access key | If Litestream is configured and `NC_DB` is not present. SQLite gets backed up to S3 |
| AWS_BUCKET | For Litestream - S3 bucket | If Litestream is configured and `NC_DB` is not present. SQLite gets backed up to S3 |
| AWS_BUCKET_PATH | For Litestream - S3 bucket path (like folder within S3 bucket) | If Litestream is configured and `NC_DB` is not present. SQLite gets backed up to S3 |
| NC_SMTP_FROM | For SMTP plugin - Email sender address | |
| NC_SMTP_HOST | For SMTP plugin - SMTP host value | |
| NC_SMTP_PORT | For SMTP plugin - SMTP port value | |
| NC_SMTP_USERNAME | For SMTP plugin (Optional) - SMTP username value for authentication | |
| NC_SMTP_PASSWORD | For SMTP plugin (Optional) - SMTP password value for authentication | |
| NC_SMTP_SECURE | For SMTP plugin (Optional) - To enable secure set value as `true` any other value treated as false | |
| NC_SMTP_IGNORE_TLS | For SMTP plugin (Optional) - To ignore tls set value as `true` any other value treated as false. For more info visit https://nodemailer.com/smtp/ | |
| NC_S3_BUCKET_NAME | For S3 storage plugin - AWS S3 bucket name | |
| NC_S3_REGION | For S3 storage plugin - AWS S3 region | |
| NC_S3_ACCESS_KEY | For S3 storage plugin - AWS access key credential for accessing resource | |
| NC_S3_ACCESS_SECRET | For S3 storage plugin - AWS access secret credential for accessing resource | |
| NC_ATTACHMENT_FIELD_SIZE | For setting the attachment field size(in Bytes) | Defaults to 20MB |
| NC_MAX_ATTACHMENTS_ALLOWED | Maximum Number of attachments per cell | |
| NC_ADMIN_EMAIL | For updating/creating super admin with provided email and password | |
| NC_ADMIN_PASSWORD | For updating/creating super admin with provided email and password. Your password should have at least 8 letters with one uppercase, one number and one special letter(Allowed special chars $&+,:;=?@#\|'.^*()%!_-" ) | |
| NODE_OPTIONS | For passing Node.js [options](https://nodejs.org/api/cli.html#node_optionsoptions) to instance | |
| NC_MINIMAL_DBS | Create a new SQLite file for each project. All the db files are stored in `nc_minimal_dbs` folder in current working directory. (This option restricts project creation on external sources) | |
| NC_DISABLE_AUDIT | Disable Audit Log | `false` |
| NC_AUTOMATION_LOG_LEVEL | Possible Values: `OFF`, `ERROR`, `ALL`. See [Webhooks](/automation/webhook/create-webhook#call-log) for details. | `OFF` |
| NC_SECURE_ATTACHMENTS | Allow accessing attachments only through presigned urls. To enable set value as `true` any other value treated as false. (⚠ this will make existing links inaccessible ⚠) | `false` |
| NC_ATTACHMENT_EXPIRE_SECONDS | How many seconds before expiring presigned attachment urls. (Attachments will expire in at least set seconds and at most 10mins after set time) | 7200 (2 hours) |
| NC_ALLOW_LOCAL_HOOKS | To enable set value as `true` any other value treated as false. (⚠ this will allow webhooks to call local links which can raise security issues ⚠) | `false` |
| NC_SANITIZE_COLUMN_NAME | Sanitize the column name during column creation. To enable set value as `true` any other value treated as false. | `true` |
| Variable | Description | If absent |
| -------- | ----------- | --------- |
| `NC_DB` | See our example database URLs [here](https://github.com/nocodb/nocodb#docker). | A local SQLite database is created in root folder if `NC_DB` is not set. |
| `NC_DB_JSON` | Can be used instead of `NC_DB` and value should be valid knex connection JSON string. | |
| `NC_DB_JSON_FILE` | Can be used instead of `NC_DB` and value should be a valid path to knex connection JSON file. | |
| `DATABASE_URL` | Can be used instead of `NC_DB` and value should be a JDBC URL string. | |
| `DATABASE_URL_FILE` | Can be used instead of `NC_DB` and value should be a valid path to a JDBC URL file. | |
| `NC_AUTH_JWT_SECRET` | JWT secret used for auth and storing other secrets. | A random secret is generated. |
| `NC_ADMIN_EMAIL` | Super admin e-mail address. | |
| `NC_ADMIN_PASSWORD` | Super admin password. The password should have at least 8 letters with one uppercase, one number and one special letter. Allowed special characters include `$&+,:;=?@#\|'.^*()%!_-"`. | |
| `PORT` | Network port NocoDB runs on. | Defaults to `8080`. |
| `DB_QUERY_LIMIT_DEFAULT` | Pagination limit. | Defaults to `25`. |
| `DB_QUERY_LIMIT_GROUP_BY_GROUP` | Group per page limit. | Defaults to `10`. |
| `DB_QUERY_LIMIT_GROUP_BY_RECORD` | Record per group limit. | Defaults to `10`. |
| `DB_QUERY_LIMIT_MAX` | Maximum allowed pagination limit. | Defaults to `1000`. |
| `DB_QUERY_LIMIT_MIN` | Minimum allowed pagination limit. | Defaults to `1`. |
| `NC_TOOL_DIR` | App directory to keep metadata and app related files in. | Defaults to the current working directory. In docker, maps to `/usr/app/data/` for mounting volume. |
| `NC_PUBLIC_URL` | Used for sending E-mail invitations. | Best guess from HTTP request params. |
| `NC_JWT_EXPIRES_IN` | JWT token expiry time | Defaults to `10h`. |
| `NC_CONNECT_TO_EXTERNAL_DB_DISABLED` | Disable base creation on external databases. | |
| `NC_MINIMAL_DBS` | Create a new SQLite file for each base. All the SQLite database files are stored in the `nc_minimal_dbs` folder in the current working directory. Enabling this option automatically sets `NC_CONNECT_TO_EXTERNAL_DB_DISABLED`, i.e. disables base creation on external databases. | |
| `NC_INVITE_ONLY_SIGNUP` | Removed since version 0.99.0, and now it's recommended to use the [super admin settings menu](/account-settings/oss-specific-details#enable--disable-signup). Disable public signup and allow signup only via invitations. | |
| `NUXT_PUBLIC_NC_BACKEND_URL` | Custom backend URL. | Defaults to `http://localhost:8080`. |
| `NC_REQUEST_BODY_SIZE` | Request body size [limit](https://expressjs.com/en/resources/middleware/body-parser.html#limit) | Defaults to `1048576`. |
| `NC_EXPORT_MAX_TIMEOUT` | After `NC_EXPORT_MAX_TIMEOUT` (in milliseconds), CSV gets downloaded in batches. | Defaults to `5000` (5 seconds). |
| `NC_DISABLE_TELE` | Disable telemetry. | |
| `NC_DASHBOARD_URL` | Custom dashboard URL path | Defaults to `/dashboard`. |
| `NC_GOOGLE_CLIENT_ID` | Google client ID to enable Google authentication. | |
| `NC_GOOGLE_CLIENT_SECRET` | Google client secret to enable Google authentication. | |
| `NC_MIGRATIONS_DISABLED` | Disable NocoDB migrations. | |
| `NC_MIN` | Disable default splash screen (initial welcome animation) and matrix screensaver. | |
| `NC_SENTRY_DSN` | Data Source Name (DSN) for Sentry monitoring. | |
| `NC_REDIS_URL` | Redis URL. Example: `redis://:authpassword@127.0.0.1:6380/4` | Meta data is stored in memory. |
| `NC_DISABLE_ERR_REPORT` | Disable error reporting. | |
| `NC_DISABLE_CACHE` | Disable cache. To be used only while debugging. If `true`, meta data is fetched from database instead of redis/cache. | Defaults to `false`. |
| `NC_SMTP_FROM` | E-mail sender address for SMTP plugin. | *SMTP plugin is disabled if this variable is not set.* |
| `NC_SMTP_HOST` | E-mail server hostname for SMTP plugin. | *SMTP plugin is disabled if this variable is not set.* |
| `NC_SMTP_PORT` | E-mail server network for SMTP plugin. | *SMTP plugin is disabled if this variable is not set.* |
| `NC_SMTP_USERNAME` | Username for authentication in SMTP plugin. | |
| `NC_SMTP_PASSWORD` | Password for authentication in SMTP plugin. | |
| `NC_SMTP_SECURE` | Enable secure authentication in SMTP plugin. Set to `true` to enable, any other value is treated as `false`. | |
| `NC_SMTP_IGNORE_TLS` | Ignore TLS in SMTP plugin. Set to `true` to ignore TLS, any other value is treated as `false`. For more information, visit [Nodemailer's SMTP documentation](https://nodemailer.com/smtp/). | |
| `NC_S3_BUCKET_NAME` | AWS S3 bucket name for S3 storage plugin. | |
| `NC_S3_REGION` | AWS S3 region for S3 storage plugin. | |
| `NC_S3_ACCESS_KEY` | AWS access key ID for S3 storage plugin. | |
| `NC_S3_ACCESS_SECRET` | AWS access secret for S3 storage plugin. | |
| `NC_ATTACHMENT_FIELD_SIZE` | Maximum file size for [attachments](/fields/field-types/custom-types/attachment/) in bytes. | Defaults to `20971520` (20 MiB). |
| `NC_MAX_ATTACHMENTS_ALLOWED` | Maximum number of attachments per cell. | Defaults to `10`. |
| `NC_SECURE_ATTACHMENTS` | Allow accessing attachments only through pre-signed URLs. Set to `true` to enable, any other value is treated as `false`. (⚠ this will make existing links inaccessible ⚠) | Defaults to `false`. |
| `NC_ATTACHMENT_EXPIRE_SECONDS` | Number of seconds after which pre-signed attachment URLs will begin to expire. The URLs will expire after `NC_ATTACHMENT_EXPIRE_SECONDS` plus 10 minutes at the very latest. | Defaults to `7200` (2 hours). |
| `NC_DISABLE_AUDIT` | Disable audit log. | Defaults to `false`. |
| `NC_AUTOMATION_LOG_LEVEL` | Possible Values: `OFF`, `ERROR`, `ALL`. See [Webhooks](/automation/webhook/create-webhook#call-log) for details. | Defaults to `OFF`. |
| `NC_ALLOW_LOCAL_HOOKS` | ⚠ Allow webhooks to call local links, which can raise security issues. ⚠ Set to `true` to enable, any other value is treated as `false` | Defaults to `false`. |
| `NC_SANITIZE_COLUMN_NAME` | Sanitize the column name during column creation. Set to `true` to enable, any other value is treated as `false` | Defaults to `true`. |
| `NODE_OPTIONS` | Node.js [options](https://nodejs.org/api/cli.html#node_optionsoptions) to pass to instance. | |
| `LITESTREAM_S3_ENDPOINT` | URL of an S3-compatible object storage service endpoint for [Litestream](https://litestream.io/) replication of NocoDB's default SQLite database. Example: `s3.eu-central-1.amazonaws.com` | *Litestream replication is disabled if this variable is not set.* |
| `LITESTREAM_S3_BUCKET` | Name of the object storage bucket to store the Litestream replication in. | *Litestream replication is disabled if this variable is not set.* |
| `LITESTREAM_S3_PATH` | Directory path to use within the Litestream replication object storage bucket. | Defaults to `nocodb`. |
| `LITESTREAM_S3_ACCESS_KEY_ID` | Authentication key ID for the Litestream replication object storage bucket. | *Litestream replication is disabled if this variable is not set.* |
| `LITESTREAM_S3_SECRET_ACCESS_KEY` | Authentication secret for the Litestream replication object storage bucket. | *Litestream replication is disabled if this variable is not set.* |
| `LITESTREAM_S3_SKIP_VERIFY` | Whether to disable TLS verification for the Litestream replication object storage service. This is useful when testing against a local node such as MinIO and you are using self-signed certificates. | Defaults to `false`. |
| `LITESTREAM_RETENTION` | Amount of time Litestream snapshot and WAL files are kept. After the retention period, a new snapshot is created and the old one is removed. WAL files that exist before the oldest snapshot will also be removed. | Defaults to `1440h` (60 days). |
| `LITESTREAM_RETENTION_CHECK_INTERVAL` | Frequency in which Litestream will check if retention needs to be enforced. | Defaults to `72h` (3 days). |
| `LITESTREAM_SNAPSHOT_INTERVAL` | Frequency in which new Litestream snapshots are created. A higher frequency reduces the time to restore since newer snapshots will have fewer WAL frames to apply. Retention still applies to these snapshots. | Defaults to `24h` (1 day). |
| `LITESTREAM_SYNC_INTERVAL` | Frequency in which frames are pushed to the Litestream replica. Increasing this frequency can increase object storage costs significantly. | Defaults to `60s` (1 minute). |

11
packages/noco-docs/docs/070.fields/040.field-types/050.custom-types/010.attachment.md

@ -62,12 +62,13 @@ Rename file only renames the file in NocoDB display (expand record & tool tip on
:::
## Environment variables
In self-hosted version, you can configure the following environment variables to customize the behavior of `Attachment` field.
- NC_ATTACHMENT_FIELD_SIZE: Max size of attachment file in bytes. Default: 20MB
- NC_SECURE_ATTACHMENTS: Allow accessing attachments only through pre-signed URLs. Default: false
- NC_ATTACHMENT_EXPIRE_SECONDS: Expiry time for pre-signed URLs. Default: 7200
In self-hosted version, you can configure the following environment variables to customize the behavior of `Attachment` field:
- `NC_ATTACHMENT_FIELD_SIZE`: Maximum size of attachment files in bytes. Default: `20971520` (20 MiB)
- `NC_MAX_ATTACHMENTS_ALLOWED`: Maximum number of attachments per cell. Default: `10`
- `NC_SECURE_ATTACHMENTS`: Allow accessing attachments only through pre-signed URLs. Default: `false`
- `NC_ATTACHMENT_EXPIRE_SECONDS`: Expiry time for pre-signed URLs. Default: `7200` (2 hours)
Find more about environment variables [here](/getting-started/self-hosted/environment-variables)
All supported environment variables are described [here](/getting-started/self-hosted/environment-variables).
## Related articles
- [Attaching a file from mobile](/views/view-types/form#attaching-a-file-from-mobile-device)

45
packages/nocodb/Dockerfile

@ -1,7 +1,9 @@
# syntax=docker/dockerfile:1
###########
# Litestream Builder
###########
FROM golang:alpine3.18 as lt-builder
FROM golang:alpine3.19 as lt-builder
WORKDIR /usr/src/
@ -9,12 +11,9 @@ RUN apk add --no-cache git make musl-dev gcc
# build litestream
RUN git clone https://github.com/benbjohnson/litestream.git litestream
RUN cd litestream ; go install ./cmd/litestream
RUN cd litestream && go install ./cmd/litestream
RUN cp $GOPATH/bin/litestream /usr/src/lt
###########
# Builder
###########
@ -28,11 +27,10 @@ RUN apk add --no-cache python3 make g++
RUN corepack enable && corepack prepare pnpm@latest --activate
# Copy application dependency manifests to the container image.
COPY ./package.json ./package.json
COPY ./docker/main.js ./docker/main.js
#COPY ./docker/start.sh /usr/src/appEntry/start.sh
COPY ./docker/start-litestream.sh /usr/src/appEntry/start.sh
COPY src/public/ ./docker/public/
COPY --link ./package.json ./package.json
COPY --link ./docker/main.js ./docker/main.js
COPY --link ./docker/start-litestream.sh /usr/src/appEntry/start.sh
COPY --link src/public/ ./docker/public/
# for pnpm to generate a flat node_modules without symlinks
# so that modclean could work as expected
@ -52,20 +50,27 @@ RUN pnpm install --prod --shamefully-hoist \
FROM alpine:3.19
WORKDIR /usr/src/app
ENV NC_DOCKER 0.6
ENV NODE_ENV production
ENV PORT 8080
ENV NC_TOOL_DIR=/usr/app/data/
RUN apk --update --no-cache add \
ENV LITESTREAM_S3_SKIP_VERIFY=false \
LITESTREAM_S3_PATH=nocodb \
LITESTREAM_RETENTION=1440h \
LITESTREAM_RETENTION_CHECK_INTERVAL=72h \
LITESTREAM_SNAPSHOT_INTERVAL=24h \
LITESTREAM_SYNC_INTERVAL=60s \
NC_DOCKER=0.6 \
NC_TOOL_DIR=/usr/app/data/ \
NODE_ENV=production \
PORT=8080
RUN apk add --update --no-cache \
nodejs \
dumb-init
# Copy litestream binary build
COPY --from=lt-builder /usr/src/lt /usr/src/appEntry/litestream
# Copy litestream binary and config file
COPY --link --from=lt-builder /usr/src/lt /usr/local/bin/litestream
COPY --link ./docker/litestream.yml /etc/litestream.yml
# Copy production code & main entry file
COPY --from=builder /usr/src/app/ /usr/src/app/
COPY --from=builder /usr/src/appEntry/ /usr/src/appEntry/
COPY --link --from=builder /usr/src/app/ /usr/src/app/
COPY --link --from=builder /usr/src/appEntry/ /usr/src/appEntry/
EXPOSE 8080
ENTRYPOINT ["/usr/bin/dumb-init", "--"]

26
packages/nocodb/Dockerfile.local

@ -1,3 +1,5 @@
# syntax=docker/dockerfile:1
###########
# Builder
###########
@ -11,11 +13,11 @@ RUN apk add --no-cache python3 make g++
RUN corepack enable && corepack prepare pnpm@latest --activate
# Copy application dependency manifests to the container image.
COPY ./package.json ./package.json
COPY ./docker/nc-gui/ ./docker/nc-gui/
COPY ./docker/main.js ./docker/index.js
COPY ./docker/start-local.sh /usr/src/appEntry/start.sh
COPY src/public/ ./docker/public/
COPY --link ./package.json ./package.json
COPY --link ./docker/nc-gui/ ./docker/nc-gui/
COPY --link ./docker/main.js ./docker/index.js
COPY --link ./docker/start-local.sh /usr/src/appEntry/start.sh
COPY --link src/public/ ./docker/public/
# for pnpm to generate a flat node_modules without symlinks
# so that modclean could work as expected
@ -36,20 +38,20 @@ RUN pnpm install --prod --shamefully-hoist --reporter=silent \
FROM alpine:3.19
WORKDIR /usr/src/app
ENV NC_DOCKER 0.6
ENV NODE_ENV production
ENV PORT 8080
ENV NC_TOOL_DIR=/usr/app/data/
ENV NC_DOCKER=0.6 \
NC_TOOL_DIR=/usr/app/data/ \
NODE_ENV=production \
PORT=8080
RUN apk --update --no-cache add \
RUN apk add --update --no-cache \
nodejs \
dumb-init \
curl \
jq
# Copy production code & main entry file
COPY --from=builder /usr/src/app/ /usr/src/app/
COPY --from=builder /usr/src/appEntry/ /usr/src/appEntry/
COPY --link --from=builder /usr/src/app/ /usr/src/app/
COPY --link --from=builder /usr/src/appEntry/ /usr/src/appEntry/
EXPOSE 8080
ENTRYPOINT ["/usr/bin/dumb-init", "--"]

22
packages/nocodb/docker/litestream.yml

@ -0,0 +1,22 @@
# Docs: https://litestream.io/reference/config/
dbs:
- path: ${NC_TOOL_DIR}noco.db
replicas:
- type: s3
endpoint: ${LITESTREAM_S3_ENDPOINT}
force-path-style: true
skip-verify: ${LITESTREAM_S3_SKIP_VERIFY}
bucket: ${LITESTREAM_S3_BUCKET}
path: ${LITESTREAM_S3_PATH}
access-key-id: ${LITESTREAM_S3_ACCESS_KEY_ID}
secret-access-key: ${LITESTREAM_S3_SECRET_ACCESS_KEY}
retention: ${LITESTREAM_RETENTION}
retention-check-interval: ${LITESTREAM_RETENTION_CHECK_INTERVAL}
snapshot-interval: ${LITESTREAM_SNAPSHOT_INTERVAL}
sync-interval: ${LITESTREAM_SYNC_INTERVAL}
# age:
# identities:
# - ${LITESTREAM_AGE_SECRET_KEY}
# recipients:
# - ${LITESTREAM_AGE_PUBLIC_KEY}

35
packages/nocodb/docker/start-litestream.sh

@ -1,26 +1,37 @@
#!/bin/sh
#sleep 5
if [ -n "${NC_TOOL_DIR}" ]; then
if [ ! -d "${NC_TOOL_DIR}" ] ; then
mkdir -p "$NC_TOOL_DIR"
fi
if [ -n "${AWS_ACCESS_KEY_ID}" ] && [ -n "${AWS_SECRET_ACCESS_KEY}" ] && [ -n "${AWS_BUCKET}" ] && [ -n "${AWS_BUCKET_PATH}" ]; then
use_litestream() {
[ -z "${NC_DB}" ] \
&& [ -z "${NC_DB_JSON}" ] \
&& [ -z "${NC_DB_JSON_FILE}" ] \
&& [ -z "${DATABASE_URL}" ] \
&& [ -z "${DATABASE_URL_FILE}" ] \
&& [ -z "${NC_MINIMAL_DBS}" ] \
&& [ -n "${LITESTREAM_S3_ENDPOINT}" ] \
&& [ -n "${LITESTREAM_S3_BUCKET}" ] \
&& [ -n "${LITESTREAM_ACCESS_KEY_ID}" ] \
&& [ -n "${LITESTREAM_SECRET_ACCESS_KEY}" ]
}
if use_litestream ; then
if [ -f "${NC_TOOL_DIR}noco.db" ]
then
if [ -f "${NC_TOOL_DIR}noco.db" ] ; then
rm "${NC_TOOL_DIR}noco.db"
rm "${NC_TOOL_DIR}noco.db-shm"
rm "${NC_TOOL_DIR}noco.db-wal"
rm -f "${NC_TOOL_DIR}noco.db-shm"
rm -f "${NC_TOOL_DIR}noco.db-wal"
fi
/usr/src/appEntry/litestream restore -o "${NC_TOOL_DIR}noco.db" "s3://$AWS_BUCKET/$AWS_BUCKET_PATH"
if [ ! -f "${NC_TOOL_DIR}noco.db" ]
then
litestream restore "${NC_TOOL_DIR}noco.db"
if [ ! -f "${NC_TOOL_DIR}noco.db" ] ; then
touch "${NC_TOOL_DIR}noco.db"
fi
/usr/src/appEntry/litestream replicate "${NC_TOOL_DIR}noco.db" "s3://$AWS_BUCKET/$AWS_BUCKET_PATH" &
litestream replicate &
fi
node docker/main.js

2
packages/nocodb/docker/start-local.sh

@ -1,6 +1,6 @@
#!/bin/sh
if [ -n "${NC_TOOL_DIR}" ]; then
if [ ! -d "${NC_TOOL_DIR}" ]; then
mkdir -p "$NC_TOOL_DIR"
fi

96
packages/nocodb/litestream/Dockerfile

@ -1,96 +0,0 @@
FROM golang:alpine3.18 as lt
WORKDIR /usr/src/
RUN apk add --no-cache git make musl-dev gcc
# build litestream
RUN git clone https://github.com/benbjohnson/litestream.git litestream
RUN cd litestream ; go install ./cmd/litestream
RUN cp $GOPATH/bin/litestream /usr/src/lt
FROM node:18.19.1-alpine as builder
WORKDIR /usr/src/app
# install pnpm
RUN corepack enable && corepack prepare pnpm@latest --activate
# Copy application dependency manifests to the container image.
COPY ./package*.json ./
COPY ./docker/main.js ./docker/main.js
#COPY ./docker/start.sh /usr/src/appEntry/start.sh
COPY ./docker/start-litestream.sh /usr/src/appEntry/start.sh
# for pnpm to generate a flat node_modules without symlinks
# so that modclean could work as expected
RUN echo "node-linker=hoisted" > .npmrc
# install production dependencies,
# reduce node_module size with modclean & removing sqlite deps,
# and add execute permission to start.sh
RUN pnpm install --prod --shamefully-hoist --reporter=silent
RUN pnpm dlx modclean --patterns="default:*" --ignore="nc-lib-gui/**,dayjs/**,express-status-monitor/**" --run
RUN rm -rf ./node_modules/sqlite3/deps
RUN chmod +x /usr/src/appEntry/start.sh
FROM alpine:3.19
#ENV AWS_ACCESS_KEY_ID=
#ENV AWS_SECRET_ACCESS_KEY=
#ENV AWS_BUCKET=
#WORKDIR /usr/src/
#
## Install go lang
#RUN apk add --no-cache git make musl-dev go
#
## Configure Go
#ENV GOROOT /usr/lib/go
#ENV GOPATH /go
#ENV PATH /go/bin:$PATH
#
#RUN mkdir -p ${GOPATH}/src ${GOPATH}/bin
#
## build litestream
#
#RUN git clone https://github.com/benbjohnson/litestream.git litestream
#RUN cd litestream ; go install ./cmd/litestream
# Bug fix for segfault ( Convert PT_GNU_STACK program header into PT_PAX_FLAGS )
#RUN apk --update --no-cache add paxctl \
# && paxctl -cm $(which node)
WORKDIR /usr/src/app
ENV NC_DOCKER 0.6
ENV PORT 8080
ENV NC_TOOL_DIR=/usr/app/data/
# Copy application dependency manifests to the container image.
# A wildcard is used to ensure both package.json AND package-lock.json are copied.
# Copying this separately prevents re-running npm install on every code change.
#COPY ./build/ ./build/
#COPY ./docker/main.js ./docker/main.js
#COPY ./package.json ./
RUN apk --update --no-cache add \
nodejs \
tar
# Copy litestream binary build
COPY --from=lt /usr/src/lt /usr/src/appEntry/litestream
# Copy production code & main entry file
COPY --from=builder /usr/src/app/ /usr/src/app/
COPY --from=builder /usr/src/appEntry/ /usr/src/appEntry/
# Run the web service on container startup.
#CMD [ "node", "docker/index.js" ]
ENTRYPOINT ["sh", "/usr/src/appEntry/start.sh"]
Loading…
Cancel
Save