github
/
nocodb
mirror of https://github.com/nocodb/nocodb
1
0
Fork 0
Code Issues Releases Wiki Activity
Browse Source

feat: Postgres base as schema (#9591) 

* refactor:  introduce `NC_DISABLE_BASE_AS_PG_SCHEMA`

* refactor: change env name and create db if not exist

* refactor: review comments

* chore: add npmignore file

Signed-off-by: Pranav C <pranavxc@gmail.com>

---------

Signed-off-by: Pranav C <pranavxc@gmail.com>
pull/9592/head
Pranav C 2 months ago committed by GitHub
parent
2411da7cee
commit
61b2562f2e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 132 additions and 91 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 49
      packages/noco-docs/docs/020.getting-started/050.self-hosted/020.environment-variables.md
  2. 10
      packages/nocodb/.npmignore
  3. 63
      packages/nocodb/src/helpers/initBaseBehaviour.ts
  4. 6
      packages/nocodb/src/providers/init-meta-service.provider.ts
  5. 95
      packages/nocodb/src/services/bases.service.ts

49
packages/noco-docs/docs/020.getting-started/050.self-hosted/020.environment-variables.md
Unescape View File

@ -75,33 +75,28 @@ For production use cases, it is crucial to set all environment variables marked
| `NC_REDIS_URL` | Yes | Specifies the Redis URL used for caching. <br></br> Eg: `redis://:authpassword@127.0.0.1:6380/4` | Caching layer of backend | | `NC_REDIS_URL` | Yes | Specifies the Redis URL used for caching. <br></br> Eg: `redis://:authpassword@127.0.0.1:6380/4` | Caching layer of backend |
## Product Configuration ## Product Configuration
| Variable | Mandatory | Description | If Not Set | | Variable | Mandatory | Description | If Not Set |
| -------- | --------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------| | -------- | --------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------|
| `DB_QUERY_LIMIT_DEFAULT` | No | Default pagination limit for data tables. | Defaults to `25`. Maximum is `100` | | `DB_QUERY_LIMIT_DEFAULT` | No | Default pagination limit for data tables. | Defaults to `25`. Maximum is `100` |
| `DB_QUERY_LIMIT_GROUP_BY_GROUP` | No | Number of groups per page. | Defaults to `10`. | | `DB_QUERY_LIMIT_GROUP_BY_GROUP` | No | Number of groups per page. | Defaults to `10`. |
| `DB_QUERY_LIMIT_GROUP_BY_RECORD` | No | Number of records per group. | Defaults to `10`. | | `DB_QUERY_LIMIT_GROUP_BY_RECORD` | No | Number of records per group. | Defaults to `10`. |
| `DB_QUERY_LIMIT_MAX` | No | Maximum allowable pagination limit. | Defaults to `1000`. | | `DB_QUERY_LIMIT_MAX` | No | Maximum allowable pagination limit. | Defaults to `1000`. |
| `DB_QUERY_LIMIT_MIN` | No | Minimum allowable pagination limit. | Defaults to `10` | | `DB_QUERY_LIMIT_MIN` | No | Minimum allowable pagination limit. | Defaults to `10` |
| `NC_CONNECT_TO_EXTERNAL_DB_DISABLED` | No | Disables the ability to create bases on external databases. | | | `NC_CONNECT_TO_EXTERNAL_DB_DISABLED` | No | Disables the ability to create bases on external databases. | |
| `NC_INVITE_ONLY_SIGNUP` | No | Disables public signup; signup is possible only via invitations. Integrated into the [super admin settings menu](/account-settings/oss-specific-details#enable--disable-signup) as of version 0.99.0. | | | `NC_INVITE_ONLY_SIGNUP` | No | Disables public signup; signup is possible only via invitations. Integrated into the [super admin settings menu](/account-settings/oss-specific-details#enable--disable-signup) as of version 0.99.0. | |
| `NC_REQUEST_BODY_SIZE` | No | Maximum bytes allowed in the request body, based on [ExpressJS limits](https://expressjs.com/en/resources/middleware/body-parser.html#limit). | Defaults to `1048576` (1 MB). | | `NC_REQUEST_BODY_SIZE` | No | Maximum bytes allowed in the request body, based on [ExpressJS limits](https://expressjs.com/en/resources/middleware/body-parser.html#limit). | Defaults to `1048576` (1 MB). |
| `NC_EXPORT_MAX_TIMEOUT` | No | Sets a timeout in milliseconds for downloading CSVs in batches if not completed within this period. | Defaults to `5000` (5 seconds). | | `NC_EXPORT_MAX_TIMEOUT` | No | Sets a timeout in milliseconds for downloading CSVs in batches if not completed within this period. | Defaults to `5000` (5 seconds). |
| `NC_ALLOW_LOCAL_HOOKS` | No | Allows webhooks to call local network links, posing potential security risks. Set to `true` to enable; all other values are considered `false`. | Defaults to `false`. | | `NC_ALLOW_LOCAL_HOOKS` | No | Allows webhooks to call local network links, posing potential security risks. Set to `true` to enable; all other values are considered `false`. | Defaults to `false`. |
| `NC_SANITIZE_COLUMN_NAME` | No | Enables sanitization of column names during their creation to prevent SQL injection and other security issues. | Defaults to `true`. | | `NC_SANITIZE_COLUMN_NAME` | No | Enables sanitization of column names during their creation to prevent SQL injection and other security issues. | Defaults to `true`. |
| `NC_TOOL_DIR` | No | Specifies the directory to store metadata and app-related files. In Docker setups, this maps to `/usr/app/data/` for mounting volumes. | Defaults to the current working directory. | | `NC_TOOL_DIR` | No | Specifies the directory to store metadata and app-related files. In Docker setups, this maps to `/usr/app/data/` for mounting volumes. | Defaults to the current working directory. |
| `NC_MINIMAL_DBS` | No | Enables the minimal database feature of NocoDB. For more details, see [Minimal Database behavior](#minimal-database). | Enabled by default for PostgreSQL when the database user has schema creation permission. Set to `false` to disable. | | `NC_DISABLE_BASE_AS_PG_SCHEMA` | No | Disables the creation of a schema for each base in PostgreSQL. [Click here for more detail](#postgres-base-as-schema) | |
| `NC_MIGRATIONS_DISABLED` | No | Disables NocoDB migrations. | | | `NC_MIGRATIONS_DISABLED` | No | Disables NocoDB migrations. | |
| `NC_DISABLE_AUDIT` | No | Disables the audit log feature. | Defaults to `false`. | | `NC_DISABLE_AUDIT` | No | Disables the audit log feature. | Defaults to `false`. |
| `NC_AUTOMATION_LOG_LEVEL` | No | Configures logging levels for automation features. Possible values: `OFF`, `ERROR`, `ALL`. More details can be found under [Webhooks](/automation/webhook/create-webhook). | Defaults to `OFF`. | | `NC_AUTOMATION_LOG_LEVEL` | No | Configures logging levels for automation features. Possible values: `OFF`, `ERROR`, `ALL`. More details can be found under [Webhooks](/automation/webhook/create-webhook). | Defaults to `OFF`. |
### Minimal Database ### Postgres Base as Schema
This feature organizes base data into smaller, modular structures for PostgreSQL and SQLite databases: For PostgreSQL, a unique schema is created for each base, providing logical separation within the database. This feature is enabled by default if the user has the required permissions. To disable it, set the `NC_DISABLE_BASE_AS_PG_SCHEMA` environment variable to `false`.
- **SQLite**: Each base's data is stored as a separate SQLite file, ensuring isolated storage.
- **PostgreSQL**: A unique schema is created for each base, providing logical separation within the database.
**Note**: For PostgreSQL, this feature is enabled by default if the user has the required permissions. To disable it, set the `NC_MINIMAL_DBS` environment variable to `false`.
## Logging & Monitoring ## Logging & Monitoring

10
packages/nocodb/.npmignore
Unescape View File

@ -0,0 +1,10 @@
node_modules
src
tests
test
docker
build-utils
webpack.*.js
tsconfig*.json
Dockerfile*
nest-cli.json

63
packages/nocodb/src/helpers/initBaseBehaviour.ts
Unescape View File

@ -3,8 +3,43 @@ import { Knex } from 'knex';
import PgConnectionConfig = Knex.PgConnectionConfig; import PgConnectionConfig = Knex.PgConnectionConfig;
import NcConnectionMgrv2 from '~/utils/common/NcConnectionMgrv2'; import NcConnectionMgrv2 from '~/utils/common/NcConnectionMgrv2';
import CustomKnex from '~/db/CustomKnex'; import CustomKnex from '~/db/CustomKnex';
import { SqlClientFactory } from '~/db/sql-client/lib/SqlClientFactory';
const logger = new Logger('initBaseBehavior'); const logger = new Logger('initBaseBehavior');
async function isSchemaCreateAllowed(
tempConnection: Knex<any, unknown[]>,
dataConfig: any,
skipDatabaseCreation = false,
) {
try {
// check if database user have permission to create new schema
return await tempConnection.raw(
"SELECT has_database_privilege(:user, :database, 'CREATE') as has_database_privilege",
{
database: (dataConfig.connection as PgConnectionConfig).database,
user: (dataConfig.connection as PgConnectionConfig).user,
},
);
} catch (e) {
// check if error is due to database not found
if (e.message.includes('does not exist') && !skipDatabaseCreation) {
// create sqlClient from dataConfig
const sqlClient = SqlClientFactory.create({
...dataConfig,
knex: tempConnection,
});
// create database if it doesn't exist
await sqlClient.createDatabaseIfNotExists({
database: dataConfig.connection.database,
schema: dataConfig.searchPath?.[0] || 'public',
});
return isSchemaCreateAllowed(tempConnection, dataConfig, true);
}
throw e;
}
}
export async function initBaseBehavior() { export async function initBaseBehavior() {
const dataConfig = await NcConnectionMgrv2.getDataConfig(); const dataConfig = await NcConnectionMgrv2.getDataConfig();
@ -13,8 +48,8 @@ export async function initBaseBehavior() {
return; return;
} }
// if NC_MINIMAL_DBS already exists, return // disable minimal databases feature if NC_DISABLE_BASE_AS_PG_SCHEMA is set to true
if (process.env.NC_MINIMAL_DBS === 'false') { if (process.env.NC_DISABLE_BASE_AS_PG_SCHEMA === 'true') {
return; return;
} }
@ -22,34 +57,30 @@ export async function initBaseBehavior() {
try { try {
tempConnection = CustomKnex(dataConfig); tempConnection = CustomKnex(dataConfig);
const schemaCreateAllowed = await isSchemaCreateAllowed(
// check if database user have permission to create new schema tempConnection,
const schemaCreateAllowed = await tempConnection.raw( dataConfig,
"SELECT has_database_privilege(:user, :database, 'CREATE') as has_database_privilege",
{
database: (dataConfig.connection as PgConnectionConfig).database,
user: (dataConfig.connection as PgConnectionConfig).user,
},
); );
// if schema creation is not allowed, return // if schema creation is not allowed, return
if (!schemaCreateAllowed.rows[0]?.has_database_privilege) { if (!schemaCreateAllowed?.rows?.[0]?.has_database_privilege) {
// set NC_MINIMAL_DBS to false if it's set to true and log warning // set NC_MINIMAL_DBS to false if it's set to true and log warning
if (process.env.NC_MINIMAL_DBS === 'true') { process.env.NC_DISABLE_BASE_AS_PG_SCHEMA = 'true';
process.env.NC_MINIMAL_DBS = 'false';
}
logger.warn( logger.warn(
`User ${(dataConfig.connection as PgConnectionConfig)?.user} does not have permission to create schema, minimal databases feature will be disabled`, `User ${
(dataConfig.connection as PgConnectionConfig)?.user
} does not have permission to create schema, minimal databases feature will be disabled`,
); );
return; return;
} }
// set NC_MINIMAL_DBS to true // set NC_MINIMAL_DBS to true
process.env.NC_MINIMAL_DBS = 'true'; process.env.NC_DISABLE_BASE_AS_PG_SCHEMA = 'false';
} catch (error) { } catch (error) {
logger.warn( logger.warn(
`Error while checking schema creation permission: ${error.message}`, `Error while checking schema creation permission: ${error.message}`,
); );
process.env.NC_DISABLE_BASE_AS_PG_SCHEMA = 'true';
} finally { } finally {
// close the connection since it's only used to verify permission // close the connection since it's only used to verify permission
await tempConnection?.destroy(); await tempConnection?.destroy();

6
packages/nocodb/src/providers/init-meta-service.provider.ts
Unescape View File

@ -84,9 +84,6 @@ export const InitMetaServiceProvider: FactoryProvider = {
Noco.config = config; Noco.config = config;
Noco.eventEmitter = eventEmitter; Noco.eventEmitter = eventEmitter;
// decide base behavior based on env and database permissions
await initBaseBehavior();
if (!instanceConfig) { if (!instanceConfig) {
// bump to latest version for fresh install // bump to latest version for fresh install
await updateMigrationJobsState({ await updateMigrationJobsState({
@ -116,6 +113,9 @@ export const InitMetaServiceProvider: FactoryProvider = {
}); });
T.emit('evt_app_started', await User.count()); T.emit('evt_app_started', await User.count());
// decide base behavior based on env and database permissions
await initBaseBehavior();
return metaService; return metaService;
}, },
provide: MetaService, provide: MetaService,

95
packages/nocodb/src/services/bases.service.ts
Unescape View File

@ -162,57 +162,62 @@ export class BasesService {
const ranId = nanoid(); const ranId = nanoid();
baseBody.prefix = `nc_${ranId}__`; baseBody.prefix = `nc_${ranId}__`;
baseBody.is_meta = true; baseBody.is_meta = true;
if (process.env.NC_MINIMAL_DBS === 'true') { const dataConfig = await Noco.getConfig()?.meta?.db;
const dataConfig = await Noco.getConfig()?.meta?.db;
if (dataConfig?.client === 'pg') { if (
baseBody.prefix = ''; dataConfig?.client === 'pg' &&
baseBody.sources = [ process.env.NC_DISABLE_BASE_AS_PG_SCHEMA !== 'true'
{ ) {
type: 'pg', baseBody.prefix = '';
is_local: true, baseBody.sources = [
is_meta: false, {
config: { type: 'pg',
schema: baseId, is_local: true,
}, is_meta: false,
inflection_column: 'camelize', config: {
inflection_table: 'camelize', schema: baseId,
}, },
]; inflection_column: 'camelize',
} else { inflection_table: 'camelize',
// if env variable NC_MINIMAL_DBS is set, then create a SQLite file/connection for each base },
// each file will be named as nc_<random_id>.db ];
const fs = require('fs'); } else if (
const toolDir = getToolDir(); dataConfig?.client === 'sqlite3' &&
const nanoidv2 = customAlphabet( process.env.NC_MINIMAL_DBS === 'true'
'1234567890abcdefghijklmnopqrstuvwxyz', ) {
14, // if env variable NC_MINIMAL_DBS is set, then create a SQLite file/connection for each base
); // each file will be named as nc_<random_id>.db
if (!(await promisify(fs.exists)(`${toolDir}/nc_minimal_dbs`))) { const fs = require('fs');
await promisify(fs.mkdir)(`${toolDir}/nc_minimal_dbs`); const toolDir = getToolDir();
} const nanoidv2 = customAlphabet(
const dbId = nanoidv2(); '1234567890abcdefghijklmnopqrstuvwxyz',
const baseTitle = DOMPurify.sanitize(baseBody.title); 14,
baseBody.prefix = ''; );
baseBody.sources = [ if (!(await promisify(fs.exists)(`${toolDir}/nc_minimal_dbs`))) {
{ await promisify(fs.mkdir)(`${toolDir}/nc_minimal_dbs`);
type: 'sqlite3', }
is_meta: false, const dbId = nanoidv2();
is_local: true, const baseTitle = DOMPurify.sanitize(baseBody.title);
config: { baseBody.prefix = '';
baseBody.sources = [
{
type: 'sqlite3',
is_meta: false,
is_local: true,
config: {
client: 'sqlite3',
connection: {
client: 'sqlite3', client: 'sqlite3',
database: baseTitle,
connection: { connection: {
client: 'sqlite3', filename: `${toolDir}/nc_minimal_dbs/${baseTitle}_${dbId}.db`,
database: baseTitle,
connection: {
filename: `${toolDir}/nc_minimal_dbs/${baseTitle}_${dbId}.db`,
},
}, },
}, },
inflection_column: 'camelize',
inflection_table: 'camelize',
}, },
]; inflection_column: 'camelize',
} inflection_table: 'camelize',
},
];
} else { } else {
const db = Noco.getConfig().meta?.db; const db = Noco.getConfig().meta?.db;
baseBody.sources = [ baseBody.sources = [

Write Preview
Loading…
Cancel
Save