Popular New Releases in Data Migration
gh-ost
GA release v1.1.4
migrations
3.5.0
phinx
0.12.10
goose
v3.5.3
mara-pipelines
Version 3.1.1
Popular Libraries in Data Migration
by github 
go
9676 MIT
GitHub's Online Schema Migrations for MySQL
by pentaho java
5719 Apache-2.0
Pentaho Data Integration ( ETL ) a.k.a Kettle
by airbnb typescript
4329 MIT
A tool to help migrate JavaScript code quickly and conveniently to TypeScript
by doctrine php
4309 MIT
Doctrine Database Migrations Library
by cakephp php
4266 MIT
PHP Database Migrations for Everyone
by Xethron php
3229 MIT
Laravel Migrations Generator: Automatically generate your migrations from an existing database schema.
by rudderlabs go
3067 AGPL-3.0
Privacy and Security focused Segment-alternative, in Golang and React
by pressly go
2481 NOASSERTION
A database migration tool. Supports SQL migrations and Go functions.
by djrobstep python
2441 Unlicense
Like diff but for PostgreSQL schemas
Trending New libraries in Data Migration
by kitloong php
924 MIT
Laravel Migrations Generator: Automatically generate your migrations from an existing database schema.
by halvardssm typescript
471 MIT
A modular Deno library for PostgreSQL, MySQL, MariaDB and SQLite migrations
by tortoise python
450 Apache-2.0
A database migrations tool for TortoiseORM, ready to production.
by adamchainz python
313 MIT
Automatically upgrade your Django projects.
by bennett-treptow php
287 MIT
Generate migrations from existing database structures
by AlmaLinux shell
145 GPL-3.0
EL to AlmaLinux migration tool.
by flow-php php
124 MIT
PHP - ETL (Extract Transform Load) data processing library
by renatootescu python
105 MIT
Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.
by ajthinking css
95
A visual process builder
Top Authors in Data Migration
1
15 Libraries
840
2
6 Libraries
43
3
4 Libraries
22
4
4 Libraries
38
5
4 Libraries
70
6
3 Libraries
115
7
3 Libraries
55
8
3 Libraries
18
9
3 Libraries
2039
10
3 Libraries
11
1
15 Libraries
840
2
6 Libraries
43
3
4 Libraries
22
4
4 Libraries
38
5
4 Libraries
70
6
3 Libraries
115
7
3 Libraries
55
8
3 Libraries
18
9
3 Libraries
2039
10
3 Libraries
11
Trending Kits in Data Migration
No Trending Kits are available at this moment for Data Migration
Trending Discussions on Data Migration
Download DML and DDL sql from Teradata using python
Cosmos DB REPLACE strings in nested Object
FastAPI refuses to let me create a mongoengine document
How to deal with missing columns on old data migrations?
Alternative Syntax or Logic for AngularFireStore Update?
How to maintain a table of all proxy models in Django?
How to abandon a manual Room migration script, and fall back to destructive migration?
Copy SQL Server Scheduled Jobs to Oracle Scheduler
Using SchemaCrawler API to get Weak/Implied Relationships
MySQL idempotent version of add column failing
QUESTION
Download DML and DDL sql from Teradata using python
Asked 2022-Mar-23 at 11:14What approach should I follow to download DDL, DML and Stored Procedures from the teradata database using python.
I have created the sample code but what is the approach to download these sql files for data migration process.
1udaExec = teradata.UdaExec(appName="HelloWorld", version="1.0",logConsole=False)
2session = udaExec.connect(method="odbc", system="xxx",username="xxx", password="xxx");
3for row in session.execute("show tables {} > {}".format(tables, export_tables)):
4 print(row)
5Unlike MSSQL which had mssql-scripter to download .sql files, does teradata provide any such option to download. Also, does it provide support to download sequences, views and procedures ?
For the Schema Migration process, what should be the best approach to download these files from the teradata as a source ?
ANSWER
Answered 2022-Mar-23 at 11:14Happy to share that I got the solution for this approach. In order to get the files in sql format use the given code to extract DDL and DML Code.
The given code is for sample database dbc.
1udaExec = teradata.UdaExec(appName="HelloWorld", version="1.0",logConsole=False)
2session = udaExec.connect(method="odbc", system="xxx",username="xxx", password="xxx");
3for row in session.execute("show tables {} > {}".format(tables, export_tables)):
4 print(row)
5with teradatasql.connect(host='enter_host_ip', user='---', password='---') as connect:
6#get the table name and database name in csv file using select statement
7df = pd.read_csv("result.csv", index_col=None)
8for tables_name in df['TableName']:
9 query = "SHOW TABLE DBC."+ tables_name
10 try:
11 df = pd.read_sql(query, connect)
12 df1 = df['Request Text'][0]
13 writePath = "C:\\Users\\SQL\\"+tables_name+".sql"
14 with open(writePath, 'a') as f:
15 dfAsString = df1
16 f.write(dfAsString)
17 except Exception as e1:
18 print(tables_name)
19 pass
20Note : Out of 192 tables I was able to get DDL/DML scripts for 189 tables. For tables perform manual intervention.
QUESTION
Cosmos DB REPLACE strings in nested Object
Asked 2022-Mar-22 at 15:09I'm using the Cosmos Data migration tool to migrate data between environments. During my migration, I need to update the hostname of the website in the data. I was able to do this pretty easily with a query like this with the top level object data:
1SELECT Farms["name"], Farms["farmerInfo"], REPLACE(Farms["websiteLink"], "thiswebsite", "newHostName") AS websiteLink FROM Farms
2My Cosmos DB data is structured like (data is just for the example):
1SELECT Farms["name"], Farms["farmerInfo"], REPLACE(Farms["websiteLink"], "thiswebsite", "newHostName") AS websiteLink FROM Farms
2{
3 "name": "Red's Farm",
4 "websiteLink": "www.thiswebsite.com/goats/",
5 "farmerInfo": {
6 "name": "Bob",
7 "websiteLink": "www.thiswebsite.com/goats/",
8 "hasGoats": true,
9 "numGoats": 17
10
11 }
12}
13I don't actually need to modify any of the top level data. The data I need to modify is within the "farmerInfo" object. I've tried a few things but I've had no luck. How can I replace a string in this object using the SQL api?
I want the data to look like this after the migration:
1SELECT Farms["name"], Farms["farmerInfo"], REPLACE(Farms["websiteLink"], "thiswebsite", "newHostName") AS websiteLink FROM Farms
2{
3 "name": "Red's Farm",
4 "websiteLink": "www.thiswebsite.com/goats/",
5 "farmerInfo": {
6 "name": "Bob",
7 "websiteLink": "www.thiswebsite.com/goats/",
8 "hasGoats": true,
9 "numGoats": 17
10
11 }
12}
13{
14 "name": "Red's Farm",
15 "websiteLink": "www.thiswebsite.com/goats/",
16 "farmerInfo": {
17 "name": "Bob",
18 "websiteLink": "www.newHostName.com/goats/", <--- Updated data
19 "hasGoats": true,
20 "numGoats": 17
21
22 }
23 }
24ANSWER
Answered 2022-Mar-22 at 15:09You can use a SELECT statement in your SELECT statement to build up the sub objects. As example:
1SELECT Farms["name"], Farms["farmerInfo"], REPLACE(Farms["websiteLink"], "thiswebsite", "newHostName") AS websiteLink FROM Farms
2{
3 "name": "Red's Farm",
4 "websiteLink": "www.thiswebsite.com/goats/",
5 "farmerInfo": {
6 "name": "Bob",
7 "websiteLink": "www.thiswebsite.com/goats/",
8 "hasGoats": true,
9 "numGoats": 17
10
11 }
12}
13{
14 "name": "Red's Farm",
15 "websiteLink": "www.thiswebsite.com/goats/",
16 "farmerInfo": {
17 "name": "Bob",
18 "websiteLink": "www.newHostName.com/goats/", <--- Updated data
19 "hasGoats": true,
20 "numGoats": 17
21
22 }
23 }
24SELECT
25 c.name,
26 c.websiteLink,
27 (
28 SELECT
29 c.farmerInfo.name,
30 REPLACE(c.farmerInfo.websiteLink, "thiswebsite", "newHostName") AS websiteLink
31 ) AS farmerInfo
32FROM c
33QUESTION
FastAPI refuses to let me create a mongoengine document
Asked 2022-Mar-10 at 02:01I have FastAPI Python application with routes that operate on a MongoDB instance. The connection works fine, and I can query documents for my GET endpoints, but creating a new document from within FastAPI seems impossible.
I consistently get:
You have not defined a default connection
I have a standalone script that handles some data migration tasks and it uses the exact same DB class and Document models that the FastAPI app does, and that script is able to save documents to mongo perfectly fine. There is no difference in how the DB object is instantiated between the API and the script.
The DB class:
1from os import getenv
2
3from mongoengine import connect
4from pymongo import MongoClient
5from pymongo.errors import ServerSelectionTimeoutError
6
7class Mongo:
8
9 @property
10 def target_db(self):
11 return 'some_db'
12
13 @property
14 def uri(self) -> str:
15 env_uri = getenv('MONGODB', None)
16 if env_uri is None:
17 raise DBError('MONGODB environment variable missing')
18 return env_uri.strip()
19
20 def connect(self) -> MongoClient:
21 try:
22 return connect(host=self.uri, db=self.target_db, alias=self.target_db)
23 except ServerSelectionTimeoutError as e:
24 raise ServerSelectionTimeoutError(e)
25All of my DB models have meta attributes defining exactly what DB and collection to use:
1from os import getenv
2
3from mongoengine import connect
4from pymongo import MongoClient
5from pymongo.errors import ServerSelectionTimeoutError
6
7class Mongo:
8
9 @property
10 def target_db(self):
11 return 'some_db'
12
13 @property
14 def uri(self) -> str:
15 env_uri = getenv('MONGODB', None)
16 if env_uri is None:
17 raise DBError('MONGODB environment variable missing')
18 return env_uri.strip()
19
20 def connect(self) -> MongoClient:
21 try:
22 return connect(host=self.uri, db=self.target_db, alias=self.target_db)
23 except ServerSelectionTimeoutError as e:
24 raise ServerSelectionTimeoutError(e)
25class Thing(Document):
26 meta = {'db_alias': 'some_db',
27 'collection': 'things'}
28Queries on existing documents succeed inside of a route definition:
1from os import getenv
2
3from mongoengine import connect
4from pymongo import MongoClient
5from pymongo.errors import ServerSelectionTimeoutError
6
7class Mongo:
8
9 @property
10 def target_db(self):
11 return 'some_db'
12
13 @property
14 def uri(self) -> str:
15 env_uri = getenv('MONGODB', None)
16 if env_uri is None:
17 raise DBError('MONGODB environment variable missing')
18 return env_uri.strip()
19
20 def connect(self) -> MongoClient:
21 try:
22 return connect(host=self.uri, db=self.target_db, alias=self.target_db)
23 except ServerSelectionTimeoutError as e:
24 raise ServerSelectionTimeoutError(e)
25class Thing(Document):
26 meta = {'db_alias': 'some_db',
27 'collection': 'things'}
28results = Thing.objects.filter(**query)
29# This returns things that I can iterate over
30Document creation fails inside of a route definition:
1from os import getenv
2
3from mongoengine import connect
4from pymongo import MongoClient
5from pymongo.errors import ServerSelectionTimeoutError
6
7class Mongo:
8
9 @property
10 def target_db(self):
11 return 'some_db'
12
13 @property
14 def uri(self) -> str:
15 env_uri = getenv('MONGODB', None)
16 if env_uri is None:
17 raise DBError('MONGODB environment variable missing')
18 return env_uri.strip()
19
20 def connect(self) -> MongoClient:
21 try:
22 return connect(host=self.uri, db=self.target_db, alias=self.target_db)
23 except ServerSelectionTimeoutError as e:
24 raise ServerSelectionTimeoutError(e)
25class Thing(Document):
26 meta = {'db_alias': 'some_db',
27 'collection': 'things'}
28results = Thing.objects.filter(**query)
29# This returns things that I can iterate over
30new_thing = Thing(**creation_args)
31new_thing.save()
32Error:
1from os import getenv
2
3from mongoengine import connect
4from pymongo import MongoClient
5from pymongo.errors import ServerSelectionTimeoutError
6
7class Mongo:
8
9 @property
10 def target_db(self):
11 return 'some_db'
12
13 @property
14 def uri(self) -> str:
15 env_uri = getenv('MONGODB', None)
16 if env_uri is None:
17 raise DBError('MONGODB environment variable missing')
18 return env_uri.strip()
19
20 def connect(self) -> MongoClient:
21 try:
22 return connect(host=self.uri, db=self.target_db, alias=self.target_db)
23 except ServerSelectionTimeoutError as e:
24 raise ServerSelectionTimeoutError(e)
25class Thing(Document):
26 meta = {'db_alias': 'some_db',
27 'collection': 'things'}
28results = Thing.objects.filter(**query)
29# This returns things that I can iterate over
30new_thing = Thing(**creation_args)
31new_thing.save()
32mongoengine.connection.ConnectionFailure: You have not defined a default connection
33What does that even mean? I know that I'm connected because I can query the db.
How is it possible that I can successfully query documents from Mongo but not save them?
Every suggestion I have seen online points to not having defined a db or alias in the call to mongoengine.connect, but I clearly am in my Mongo object, and even if that were true, surely I wouldn't be able to retrieve documents from the db collection...
ANSWER
Answered 2022-Mar-10 at 02:01The mongoengine Document models had malformatted meta attributes...
1from os import getenv
2
3from mongoengine import connect
4from pymongo import MongoClient
5from pymongo.errors import ServerSelectionTimeoutError
6
7class Mongo:
8
9 @property
10 def target_db(self):
11 return 'some_db'
12
13 @property
14 def uri(self) -> str:
15 env_uri = getenv('MONGODB', None)
16 if env_uri is None:
17 raise DBError('MONGODB environment variable missing')
18 return env_uri.strip()
19
20 def connect(self) -> MongoClient:
21 try:
22 return connect(host=self.uri, db=self.target_db, alias=self.target_db)
23 except ServerSelectionTimeoutError as e:
24 raise ServerSelectionTimeoutError(e)
25class Thing(Document):
26 meta = {'db_alias': 'some_db',
27 'collection': 'things'}
28results = Thing.objects.filter(**query)
29# This returns things that I can iterate over
30new_thing = Thing(**creation_args)
31new_thing.save()
32mongoengine.connection.ConnectionFailure: You have not defined a default connection
33class Accounts(Document):
34 meta = {'db_alias': 'some_db',
35 'collection': 'things'}
36Solution: db_alias needed to be changed to db.
I didn't find this through documentation, and definitely not through the extremely unhelpful error messages. I just tried it on a whim. Now everything works using the FastAPI framework.
QUESTION
How to deal with missing columns on old data migrations?
Asked 2022-Mar-09 at 02:13I want to add a new column to a django model. However, I have a past data migration that creates an object of this model, which now doesn't work because the column didn't exist at the time of that migration.
i.e. my model is:
1class MyModel(models.Model):
2 name = models.CharField()
3 foo = models.IntegerField() # <-- I want to add this column
4however my old migration is this:
1class MyModel(models.Model):
2 name = models.CharField()
3 foo = models.IntegerField() # <-- I want to add this column
4def add_my_model(apps, schema_editor):
5 MyModel.objects.create(name="blah")
6
7class Migration(migrations.Migration):
8 # ...
9 operations = [
10 migrations.RunPython(add_my_model),
11 ]
12And when I try to run migrations on a fresh db, this migration fails with an error similar to this:
1class MyModel(models.Model):
2 name = models.CharField()
3 foo = models.IntegerField() # <-- I want to add this column
4def add_my_model(apps, schema_editor):
5 MyModel.objects.create(name="blah")
6
7class Migration(migrations.Migration):
8 # ...
9 operations = [
10 migrations.RunPython(add_my_model),
11 ]
12query = 'INSERT INTO "my_model" ("name", "foo") VALUES (?, ?)'
13params = ['blah', None]
14
15...
16
17E sqlite3.OperationalError: table my_model has no column named foo
18What's the best way to deal with this situation?
ANSWER
Answered 2022-Mar-09 at 02:13This is explained in Django documentation:
1class MyModel(models.Model):
2 name = models.CharField()
3 foo = models.IntegerField() # <-- I want to add this column
4def add_my_model(apps, schema_editor):
5 MyModel.objects.create(name="blah")
6
7class Migration(migrations.Migration):
8 # ...
9 operations = [
10 migrations.RunPython(add_my_model),
11 ]
12query = 'INSERT INTO "my_model" ("name", "foo") VALUES (?, ?)'
13params = ['blah', None]
14
15...
16
17E sqlite3.OperationalError: table my_model has no column named foo
18def combine_names(apps, schema_editor):
19 # We can't import the Person model directly as it may be a newer
20 # version than this migration expects. We use the historical version.
21 Person = apps.get_model('yourappname', 'Person')
22basically instead of directly importing MyModel you should be doing something like
1class MyModel(models.Model):
2 name = models.CharField()
3 foo = models.IntegerField() # <-- I want to add this column
4def add_my_model(apps, schema_editor):
5 MyModel.objects.create(name="blah")
6
7class Migration(migrations.Migration):
8 # ...
9 operations = [
10 migrations.RunPython(add_my_model),
11 ]
12query = 'INSERT INTO "my_model" ("name", "foo") VALUES (?, ?)'
13params = ['blah', None]
14
15...
16
17E sqlite3.OperationalError: table my_model has no column named foo
18def combine_names(apps, schema_editor):
19 # We can't import the Person model directly as it may be a newer
20 # version than this migration expects. We use the historical version.
21 Person = apps.get_model('yourappname', 'Person')
22def add_my_model(apps, schema_editor):
23 MyModel = apps.get_model('yourappname', 'MyModel')
24 MyModel.objects.create(name="blah")
25QUESTION
Alternative Syntax or Logic for AngularFireStore Update?
Asked 2022-Mar-07 at 12:45I would like to ask for alternative ways to implement Update data in firestore inside a component ts.
Normally I would use a service and subscribe thru it, however, subscribing and using the service became too laggy and our app crashes since our data team injected around 20k+ records due to data migration.
Normally I do the following the old way:
1
2Via Service:
3
4
5import { Injectable } from '@angular/core';
6import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument} from '@angular/fire/compat/firestore';
7import { Observable } from 'rxjs';
8import {map } from 'rxjs/operators';
9
10@Injectable({
11 providedIn: 'root'
12})
13
14export class CustomerService {
15 customerCollection: AngularFirestoreCollection <Customer>;
16 customerDoc: AngularFirestoreDocument <Customer>;
17 customers: Observable<Customer[]>;
18
19 constructor(
20 public angularFireStore: AngularFirestore
21
22 )
23 {}
24
25 getCustomer() {
26 this.customerCollection = this.angularFireStore.collection('customer');
27 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes => {
28 return changes.map( a => {
29 const data = a.payload.doc.data() as Customer;
30 data.id=a.payload.doc.id;
31 return data;
32 })
33
34 }))
35 return this.customers;
36 }
37
38 addCustomer(customerAdd:Customer) {
39 this.customerCollection.add(customerAdd);
40 };
41
42 updateCustomer(custcol : Customer) {
43 this.customerDoc = this.angularFireStore.doc(`customer/${custcol.id}`);
44 this.customerDoc.update(custcol);
45 }
46
47
48}
49
50Normally in my Component TS Old Code, I just usually put the update on a function like this.
1
2Via Service:
3
4
5import { Injectable } from '@angular/core';
6import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument} from '@angular/fire/compat/firestore';
7import { Observable } from 'rxjs';
8import {map } from 'rxjs/operators';
9
10@Injectable({
11 providedIn: 'root'
12})
13
14export class CustomerService {
15 customerCollection: AngularFirestoreCollection <Customer>;
16 customerDoc: AngularFirestoreDocument <Customer>;
17 customers: Observable<Customer[]>;
18
19 constructor(
20 public angularFireStore: AngularFirestore
21
22 )
23 {}
24
25 getCustomer() {
26 this.customerCollection = this.angularFireStore.collection('customer');
27 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes => {
28 return changes.map( a => {
29 const data = a.payload.doc.data() as Customer;
30 data.id=a.payload.doc.id;
31 return data;
32 })
33
34 }))
35 return this.customers;
36 }
37
38 addCustomer(customerAdd:Customer) {
39 this.customerCollection.add(customerAdd);
40 };
41
42 updateCustomer(custcol : Customer) {
43 this.customerDoc = this.angularFireStore.doc(`customer/${custcol.id}`);
44 this.customerDoc.update(custcol);
45 }
46
47
48}
49
50 customerList: customerModel[];
51 customerDetails : customerModel;
52 customerName: any;
53
54 addCustomer : Customer ={
55 customerNo: '',
56 customerAccountNo: '',
57 customerName: '',
58 customerAddress: '',
59 customerContactNo: '',
60 customerEmail: ''
61
62 };
63
64ngOnInit(): void {
65 this.customerService.getCustomer().subscribe( customerObservable => {
66 this.customerList = customerObservable});
67}
68
69
70add() {
71 this.addCustomer.customerName = this.customerName //(input comes from ngModel btw)
72 this.customerCollection.add(this.addCustomer);
73}
74
75update(){
76 this.customerDetails.customerName = this.customerName //(input comes from ngModel btw)
77 this.customerService.UpdateCustomer(this.customerDetails)
78}
79This works properly and smoothly when we only have 6000+ records, but as stated above after injecting additional records loading and even updating become laggy.
Searching thru solutions, we stop using the service, and put everything inside the component instead , I manage to implement the Query (but just ValueChanges, I believe in order to update I should use SnapshotChanges) and Adding, however there's very minimal or no documentation at all for the Update even on the official AngularFireStore Documentation, that's why I'm struggling for the correct syntax/logic for this one.
Here's my updated Component TS Code
1
2Via Service:
3
4
5import { Injectable } from '@angular/core';
6import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument} from '@angular/fire/compat/firestore';
7import { Observable } from 'rxjs';
8import {map } from 'rxjs/operators';
9
10@Injectable({
11 providedIn: 'root'
12})
13
14export class CustomerService {
15 customerCollection: AngularFirestoreCollection <Customer>;
16 customerDoc: AngularFirestoreDocument <Customer>;
17 customers: Observable<Customer[]>;
18
19 constructor(
20 public angularFireStore: AngularFirestore
21
22 )
23 {}
24
25 getCustomer() {
26 this.customerCollection = this.angularFireStore.collection('customer');
27 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes => {
28 return changes.map( a => {
29 const data = a.payload.doc.data() as Customer;
30 data.id=a.payload.doc.id;
31 return data;
32 })
33
34 }))
35 return this.customers;
36 }
37
38 addCustomer(customerAdd:Customer) {
39 this.customerCollection.add(customerAdd);
40 };
41
42 updateCustomer(custcol : Customer) {
43 this.customerDoc = this.angularFireStore.doc(`customer/${custcol.id}`);
44 this.customerDoc.update(custcol);
45 }
46
47
48}
49
50 customerList: customerModel[];
51 customerDetails : customerModel;
52 customerName: any;
53
54 addCustomer : Customer ={
55 customerNo: '',
56 customerAccountNo: '',
57 customerName: '',
58 customerAddress: '',
59 customerContactNo: '',
60 customerEmail: ''
61
62 };
63
64ngOnInit(): void {
65 this.customerService.getCustomer().subscribe( customerObservable => {
66 this.customerList = customerObservable});
67}
68
69
70add() {
71 this.addCustomer.customerName = this.customerName //(input comes from ngModel btw)
72 this.customerCollection.add(this.addCustomer);
73}
74
75update(){
76 this.customerDetails.customerName = this.customerName //(input comes from ngModel btw)
77 this.customerService.UpdateCustomer(this.customerDetails)
78}
79import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument } from '@angular/fire/compat/firestore';
80import { Observable} from 'rxjs';
81import { Customer } from '../../models/customer.model';
82
83
84@Component({
85 selector: 'app-add-client',
86 templateUrl: './add-client.component.html',
87 styleUrls: ['./add-client.component.scss']
88})
89export class AddClientComponent implements OnInit {
90
91 private customerCollection: AngularFirestoreCollection<Customer>
92 private customerDocs: AngularFirestoreDocument<Customer>
93 customers: Observable<Customer[]>
94
95 constructor(
96 public afs: AngularFirestore
97 ) {
98 this.customerCollection = this.afs.collection<Customer>('customer')
99 this.customers = this.customerCollection.valueChanges();
100 }
101 customerList: Customer[];
102 customerDetails: Customer;
103
104 addCustomer : Customer ={
105 customerNo: '',
106 customerAccountNo: '',
107 customerName: '',
108 customerAddress: '',
109 customerContactNo: '',
110 customerEmail: ''
111
112 };
113
114queryCustomer() {
115
116 this.customerCollection = this.afs.collection('customer',
117 ref => ref
118 .orderBy('customerName')
119 .startAt(this.query)
120 .endAt(this.query + "\uf8ff"));
121 this.customers = this.customerCollection.valueChanges();
122 this.customers.subscribe(
123 (data) => (this.customerList = (data)));
124 }
125
126add() {
127 this.addCustomer.customerName = this.customerName;
128 this.customerCollection.add(this.addCustomer);
129}
130
131queryCustomer() {
132
133 this.customerCollection = this.afs.collection('customer',
134 ref => ref
135 .orderBy('customerName')
136 .startAt(this.query)
137 .endAt(this.query + "\uf8ff"));
138 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes =>{
139 return changes.map(a =>{
140 const data=a.payload.doc.data() as depositModel;
141 data.id=a.payload.doc.id;
142 return data;
143 });
144 }
145 ));
146 this.customers.subscribe(
147 (data) => (this.customerList = (data)));
148 }
149
150
151}
152I did the following attempts, but somehow they don't work
1
2Via Service:
3
4
5import { Injectable } from '@angular/core';
6import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument} from '@angular/fire/compat/firestore';
7import { Observable } from 'rxjs';
8import {map } from 'rxjs/operators';
9
10@Injectable({
11 providedIn: 'root'
12})
13
14export class CustomerService {
15 customerCollection: AngularFirestoreCollection <Customer>;
16 customerDoc: AngularFirestoreDocument <Customer>;
17 customers: Observable<Customer[]>;
18
19 constructor(
20 public angularFireStore: AngularFirestore
21
22 )
23 {}
24
25 getCustomer() {
26 this.customerCollection = this.angularFireStore.collection('customer');
27 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes => {
28 return changes.map( a => {
29 const data = a.payload.doc.data() as Customer;
30 data.id=a.payload.doc.id;
31 return data;
32 })
33
34 }))
35 return this.customers;
36 }
37
38 addCustomer(customerAdd:Customer) {
39 this.customerCollection.add(customerAdd);
40 };
41
42 updateCustomer(custcol : Customer) {
43 this.customerDoc = this.angularFireStore.doc(`customer/${custcol.id}`);
44 this.customerDoc.update(custcol);
45 }
46
47
48}
49
50 customerList: customerModel[];
51 customerDetails : customerModel;
52 customerName: any;
53
54 addCustomer : Customer ={
55 customerNo: '',
56 customerAccountNo: '',
57 customerName: '',
58 customerAddress: '',
59 customerContactNo: '',
60 customerEmail: ''
61
62 };
63
64ngOnInit(): void {
65 this.customerService.getCustomer().subscribe( customerObservable => {
66 this.customerList = customerObservable});
67}
68
69
70add() {
71 this.addCustomer.customerName = this.customerName //(input comes from ngModel btw)
72 this.customerCollection.add(this.addCustomer);
73}
74
75update(){
76 this.customerDetails.customerName = this.customerName //(input comes from ngModel btw)
77 this.customerService.UpdateCustomer(this.customerDetails)
78}
79import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument } from '@angular/fire/compat/firestore';
80import { Observable} from 'rxjs';
81import { Customer } from '../../models/customer.model';
82
83
84@Component({
85 selector: 'app-add-client',
86 templateUrl: './add-client.component.html',
87 styleUrls: ['./add-client.component.scss']
88})
89export class AddClientComponent implements OnInit {
90
91 private customerCollection: AngularFirestoreCollection<Customer>
92 private customerDocs: AngularFirestoreDocument<Customer>
93 customers: Observable<Customer[]>
94
95 constructor(
96 public afs: AngularFirestore
97 ) {
98 this.customerCollection = this.afs.collection<Customer>('customer')
99 this.customers = this.customerCollection.valueChanges();
100 }
101 customerList: Customer[];
102 customerDetails: Customer;
103
104 addCustomer : Customer ={
105 customerNo: '',
106 customerAccountNo: '',
107 customerName: '',
108 customerAddress: '',
109 customerContactNo: '',
110 customerEmail: ''
111
112 };
113
114queryCustomer() {
115
116 this.customerCollection = this.afs.collection('customer',
117 ref => ref
118 .orderBy('customerName')
119 .startAt(this.query)
120 .endAt(this.query + "\uf8ff"));
121 this.customers = this.customerCollection.valueChanges();
122 this.customers.subscribe(
123 (data) => (this.customerList = (data)));
124 }
125
126add() {
127 this.addCustomer.customerName = this.customerName;
128 this.customerCollection.add(this.addCustomer);
129}
130
131queryCustomer() {
132
133 this.customerCollection = this.afs.collection('customer',
134 ref => ref
135 .orderBy('customerName')
136 .startAt(this.query)
137 .endAt(this.query + "\uf8ff"));
138 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes =>{
139 return changes.map(a =>{
140 const data=a.payload.doc.data() as depositModel;
141 data.id=a.payload.doc.id;
142 return data;
143 });
144 }
145 ));
146 this.customers.subscribe(
147 (data) => (this.customerList = (data)));
148 }
149
150
151}
152
153update() {
154
155 this.customerDocs = this.angularFirestore.doc(`jobOrders/${customerDetails.id}`);
156 this.customerDocs.Update(customerDetails);
157
158}
159
160
161I'm really new to this kind of implementation and hopefully, someone could help me on this one. Thank you.
ANSWER
Answered 2022-Mar-07 at 12:451
2Via Service:
3
4
5import { Injectable } from '@angular/core';
6import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument} from '@angular/fire/compat/firestore';
7import { Observable } from 'rxjs';
8import {map } from 'rxjs/operators';
9
10@Injectable({
11 providedIn: 'root'
12})
13
14export class CustomerService {
15 customerCollection: AngularFirestoreCollection <Customer>;
16 customerDoc: AngularFirestoreDocument <Customer>;
17 customers: Observable<Customer[]>;
18
19 constructor(
20 public angularFireStore: AngularFirestore
21
22 )
23 {}
24
25 getCustomer() {
26 this.customerCollection = this.angularFireStore.collection('customer');
27 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes => {
28 return changes.map( a => {
29 const data = a.payload.doc.data() as Customer;
30 data.id=a.payload.doc.id;
31 return data;
32 })
33
34 }))
35 return this.customers;
36 }
37
38 addCustomer(customerAdd:Customer) {
39 this.customerCollection.add(customerAdd);
40 };
41
42 updateCustomer(custcol : Customer) {
43 this.customerDoc = this.angularFireStore.doc(`customer/${custcol.id}`);
44 this.customerDoc.update(custcol);
45 }
46
47
48}
49
50 customerList: customerModel[];
51 customerDetails : customerModel;
52 customerName: any;
53
54 addCustomer : Customer ={
55 customerNo: '',
56 customerAccountNo: '',
57 customerName: '',
58 customerAddress: '',
59 customerContactNo: '',
60 customerEmail: ''
61
62 };
63
64ngOnInit(): void {
65 this.customerService.getCustomer().subscribe( customerObservable => {
66 this.customerList = customerObservable});
67}
68
69
70add() {
71 this.addCustomer.customerName = this.customerName //(input comes from ngModel btw)
72 this.customerCollection.add(this.addCustomer);
73}
74
75update(){
76 this.customerDetails.customerName = this.customerName //(input comes from ngModel btw)
77 this.customerService.UpdateCustomer(this.customerDetails)
78}
79import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument } from '@angular/fire/compat/firestore';
80import { Observable} from 'rxjs';
81import { Customer } from '../../models/customer.model';
82
83
84@Component({
85 selector: 'app-add-client',
86 templateUrl: './add-client.component.html',
87 styleUrls: ['./add-client.component.scss']
88})
89export class AddClientComponent implements OnInit {
90
91 private customerCollection: AngularFirestoreCollection<Customer>
92 private customerDocs: AngularFirestoreDocument<Customer>
93 customers: Observable<Customer[]>
94
95 constructor(
96 public afs: AngularFirestore
97 ) {
98 this.customerCollection = this.afs.collection<Customer>('customer')
99 this.customers = this.customerCollection.valueChanges();
100 }
101 customerList: Customer[];
102 customerDetails: Customer;
103
104 addCustomer : Customer ={
105 customerNo: '',
106 customerAccountNo: '',
107 customerName: '',
108 customerAddress: '',
109 customerContactNo: '',
110 customerEmail: ''
111
112 };
113
114queryCustomer() {
115
116 this.customerCollection = this.afs.collection('customer',
117 ref => ref
118 .orderBy('customerName')
119 .startAt(this.query)
120 .endAt(this.query + "\uf8ff"));
121 this.customers = this.customerCollection.valueChanges();
122 this.customers.subscribe(
123 (data) => (this.customerList = (data)));
124 }
125
126add() {
127 this.addCustomer.customerName = this.customerName;
128 this.customerCollection.add(this.addCustomer);
129}
130
131queryCustomer() {
132
133 this.customerCollection = this.afs.collection('customer',
134 ref => ref
135 .orderBy('customerName')
136 .startAt(this.query)
137 .endAt(this.query + "\uf8ff"));
138 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes =>{
139 return changes.map(a =>{
140 const data=a.payload.doc.data() as depositModel;
141 data.id=a.payload.doc.id;
142 return data;
143 });
144 }
145 ));
146 this.customers.subscribe(
147 (data) => (this.customerList = (data)));
148 }
149
150
151}
152
153update() {
154
155 this.customerDocs = this.angularFirestore.doc(`jobOrders/${customerDetails.id}`);
156 this.customerDocs.Update(customerDetails);
157
158}
159
160
161onRename() {
162 if (!this.editForm.value.replaceValue) {
163 this.message = "Cannot Be Empty!";
164 } else {
165 this.firestore.collection('testCollection').doc(this.id).update({ field: this.editForm.value.replaceValue });
166 this.edit = false;
167 this.message2 = '';
168 this.single = null;
169 }
170 }
171Example2 :
1
2Via Service:
3
4
5import { Injectable } from '@angular/core';
6import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument} from '@angular/fire/compat/firestore';
7import { Observable } from 'rxjs';
8import {map } from 'rxjs/operators';
9
10@Injectable({
11 providedIn: 'root'
12})
13
14export class CustomerService {
15 customerCollection: AngularFirestoreCollection <Customer>;
16 customerDoc: AngularFirestoreDocument <Customer>;
17 customers: Observable<Customer[]>;
18
19 constructor(
20 public angularFireStore: AngularFirestore
21
22 )
23 {}
24
25 getCustomer() {
26 this.customerCollection = this.angularFireStore.collection('customer');
27 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes => {
28 return changes.map( a => {
29 const data = a.payload.doc.data() as Customer;
30 data.id=a.payload.doc.id;
31 return data;
32 })
33
34 }))
35 return this.customers;
36 }
37
38 addCustomer(customerAdd:Customer) {
39 this.customerCollection.add(customerAdd);
40 };
41
42 updateCustomer(custcol : Customer) {
43 this.customerDoc = this.angularFireStore.doc(`customer/${custcol.id}`);
44 this.customerDoc.update(custcol);
45 }
46
47
48}
49
50 customerList: customerModel[];
51 customerDetails : customerModel;
52 customerName: any;
53
54 addCustomer : Customer ={
55 customerNo: '',
56 customerAccountNo: '',
57 customerName: '',
58 customerAddress: '',
59 customerContactNo: '',
60 customerEmail: ''
61
62 };
63
64ngOnInit(): void {
65 this.customerService.getCustomer().subscribe( customerObservable => {
66 this.customerList = customerObservable});
67}
68
69
70add() {
71 this.addCustomer.customerName = this.customerName //(input comes from ngModel btw)
72 this.customerCollection.add(this.addCustomer);
73}
74
75update(){
76 this.customerDetails.customerName = this.customerName //(input comes from ngModel btw)
77 this.customerService.UpdateCustomer(this.customerDetails)
78}
79import { AngularFirestore, AngularFirestoreCollection, AngularFirestoreDocument } from '@angular/fire/compat/firestore';
80import { Observable} from 'rxjs';
81import { Customer } from '../../models/customer.model';
82
83
84@Component({
85 selector: 'app-add-client',
86 templateUrl: './add-client.component.html',
87 styleUrls: ['./add-client.component.scss']
88})
89export class AddClientComponent implements OnInit {
90
91 private customerCollection: AngularFirestoreCollection<Customer>
92 private customerDocs: AngularFirestoreDocument<Customer>
93 customers: Observable<Customer[]>
94
95 constructor(
96 public afs: AngularFirestore
97 ) {
98 this.customerCollection = this.afs.collection<Customer>('customer')
99 this.customers = this.customerCollection.valueChanges();
100 }
101 customerList: Customer[];
102 customerDetails: Customer;
103
104 addCustomer : Customer ={
105 customerNo: '',
106 customerAccountNo: '',
107 customerName: '',
108 customerAddress: '',
109 customerContactNo: '',
110 customerEmail: ''
111
112 };
113
114queryCustomer() {
115
116 this.customerCollection = this.afs.collection('customer',
117 ref => ref
118 .orderBy('customerName')
119 .startAt(this.query)
120 .endAt(this.query + "\uf8ff"));
121 this.customers = this.customerCollection.valueChanges();
122 this.customers.subscribe(
123 (data) => (this.customerList = (data)));
124 }
125
126add() {
127 this.addCustomer.customerName = this.customerName;
128 this.customerCollection.add(this.addCustomer);
129}
130
131queryCustomer() {
132
133 this.customerCollection = this.afs.collection('customer',
134 ref => ref
135 .orderBy('customerName')
136 .startAt(this.query)
137 .endAt(this.query + "\uf8ff"));
138 this.customers = this.customerCollection.snapshotChanges().pipe(map(changes =>{
139 return changes.map(a =>{
140 const data=a.payload.doc.data() as depositModel;
141 data.id=a.payload.doc.id;
142 return data;
143 });
144 }
145 ));
146 this.customers.subscribe(
147 (data) => (this.customerList = (data)));
148 }
149
150
151}
152
153update() {
154
155 this.customerDocs = this.angularFirestore.doc(`jobOrders/${customerDetails.id}`);
156 this.customerDocs.Update(customerDetails);
157
158}
159
160
161onRename() {
162 if (!this.editForm.value.replaceValue) {
163 this.message = "Cannot Be Empty!";
164 } else {
165 this.firestore.collection('testCollection').doc(this.id).update({ field: this.editForm.value.replaceValue });
166 this.edit = false;
167 this.message2 = '';
168 this.single = null;
169 }
170 }
171updateCoffeeOrder(data) {
172 return
173 this.firestore
174 .collection("coffeeOrders")
175 .doc(data.payload.doc.id)
176 .set({ completed: true }, { merge: true });
177}
178You can refer to the stackoverflow thread and video on CRUD operations.
QUESTION
How to maintain a table of all proxy models in Django?
Asked 2022-Mar-04 at 17:01I have a model A and want to make subclasses of it.
1class A(models.Model):
2 type = models.ForeignKey(Type)
3 data = models.JSONField()
4
5 def compute():
6 pass
7
8class B(A):
9 def compute():
10 df = self.go_get_data()
11 self.data = self.process(df)
12
13class C(A):
14 def compute():
15 df = self.go_get_other_data()
16 self.data = self.process_another_way(df)
17
18# ... other subclasses of A
19B and C should not have their own tables, so I decided to use the proxy attirbute of Meta. However, I want there to be a table of all the implemented proxies.
In particular, I want to keep a record of the name and description of each subclass.
For example, for B, the name would be "B" and the description would be the docstring for B.
So I made another model:
1class A(models.Model):
2 type = models.ForeignKey(Type)
3 data = models.JSONField()
4
5 def compute():
6 pass
7
8class B(A):
9 def compute():
10 df = self.go_get_data()
11 self.data = self.process(df)
12
13class C(A):
14 def compute():
15 df = self.go_get_other_data()
16 self.data = self.process_another_way(df)
17
18# ... other subclasses of A
19class Type(models.Model):
20 # The name of the class
21 name = models.String()
22 # The docstring of the class
23 desc = models.String()
24 # A unique identifier, different from the Django ID,
25 # that allows for smoothly changing the name of the class
26 identifier = models.Int()
27Now, I want it so when I create an A, I can only choose between the different subclasses of A.
Hence the Type table should always be up-to-date.
For example, if I want to unit-test the behavior of B, I'll need to use the corresponding Type instance to create an instance of B, so that Type instance already needs to be in the database.
Looking over on the Django website, I see two ways to achieve this: fixtures and data migrations. Fixtures aren't dynamic enough for my usecase, since the attributes literally come from the code. That leaves me with data migrations.
I tried writing one, that goes something like this:
1class A(models.Model):
2 type = models.ForeignKey(Type)
3 data = models.JSONField()
4
5 def compute():
6 pass
7
8class B(A):
9 def compute():
10 df = self.go_get_data()
11 self.data = self.process(df)
12
13class C(A):
14 def compute():
15 df = self.go_get_other_data()
16 self.data = self.process_another_way(df)
17
18# ... other subclasses of A
19class Type(models.Model):
20 # The name of the class
21 name = models.String()
22 # The docstring of the class
23 desc = models.String()
24 # A unique identifier, different from the Django ID,
25 # that allows for smoothly changing the name of the class
26 identifier = models.Int()
27def update_results(apps, schema_editor):
28 A = apps.get_model("app", "A")
29 Type = apps.get_model("app", "Type")
30 subclasses = get_all_subclasses(A)
31 for cls in subclasses:
32 id = cls.get_identifier()
33 Type.objects.update_or_create(
34 identifier=id,
35 defaults=dict(name=cls.__name__, desc=cls.__desc__)
36 )
37
38class Migration(migrations.Migration):
39
40 operations = [
41 RunPython(update_results)
42 ]
43
44 # ... other stuff
45The problem is, I don't see how to store the identifier within the class, so that the Django Model instance can recover it.
So far, here is what I have tried:
I have tried using the fairly new __init_subclass__ construct of Python. So my code now looks like:
1class A(models.Model):
2 type = models.ForeignKey(Type)
3 data = models.JSONField()
4
5 def compute():
6 pass
7
8class B(A):
9 def compute():
10 df = self.go_get_data()
11 self.data = self.process(df)
12
13class C(A):
14 def compute():
15 df = self.go_get_other_data()
16 self.data = self.process_another_way(df)
17
18# ... other subclasses of A
19class Type(models.Model):
20 # The name of the class
21 name = models.String()
22 # The docstring of the class
23 desc = models.String()
24 # A unique identifier, different from the Django ID,
25 # that allows for smoothly changing the name of the class
26 identifier = models.Int()
27def update_results(apps, schema_editor):
28 A = apps.get_model("app", "A")
29 Type = apps.get_model("app", "Type")
30 subclasses = get_all_subclasses(A)
31 for cls in subclasses:
32 id = cls.get_identifier()
33 Type.objects.update_or_create(
34 identifier=id,
35 defaults=dict(name=cls.__name__, desc=cls.__desc__)
36 )
37
38class Migration(migrations.Migration):
39
40 operations = [
41 RunPython(update_results)
42 ]
43
44 # ... other stuff
45class A:
46
47 def __init_subclass__(cls, identifier=None, **kwargs):
48 super().__init_subclass__(**kwargs)
49 if identifier is None:
50 raise ValueError()
51 cls.identifier = identifier
52 Type.objects.update_or_create(
53 identifier=identifier,
54 defaults=dict(name=cls.__name__, desc=cls.__doc__)
55 )
56
57 # ... the rest of A
58
59# The identifier should never change, so that even if the
60# name of the class changes, we still know which subclass is referred to
61class B(A, identifier=3):
62
63 # ... the rest of B
64But this update_or_create fails when the database is new (e.g. during unit tests), because the Type table does not exist.
When I have this problem in development (we're still in early stages so deleting the DB is still sensible), I have to go
comment out the update_or_create in __init_subclass__. I can then migrate and put it back in.
Of course, this solution is also not great because __init_subclass__ is run way more than necessary. Ideally this machinery would only happen at migration.
So there you have it! I hope the problem statement makes sense.
Thanks for reading this far and I look forward to hearing from you; even if you have other things to do, I wish you a good rest of your day :)
ANSWER
Answered 2022-Mar-04 at 17:01With a little help from Django-expert friends, I solved this with the post_migrate signal.
I removed the update_or_create in __init_subclass, and in project/app/apps.py I added:
1class A(models.Model):
2 type = models.ForeignKey(Type)
3 data = models.JSONField()
4
5 def compute():
6 pass
7
8class B(A):
9 def compute():
10 df = self.go_get_data()
11 self.data = self.process(df)
12
13class C(A):
14 def compute():
15 df = self.go_get_other_data()
16 self.data = self.process_another_way(df)
17
18# ... other subclasses of A
19class Type(models.Model):
20 # The name of the class
21 name = models.String()
22 # The docstring of the class
23 desc = models.String()
24 # A unique identifier, different from the Django ID,
25 # that allows for smoothly changing the name of the class
26 identifier = models.Int()
27def update_results(apps, schema_editor):
28 A = apps.get_model("app", "A")
29 Type = apps.get_model("app", "Type")
30 subclasses = get_all_subclasses(A)
31 for cls in subclasses:
32 id = cls.get_identifier()
33 Type.objects.update_or_create(
34 identifier=id,
35 defaults=dict(name=cls.__name__, desc=cls.__desc__)
36 )
37
38class Migration(migrations.Migration):
39
40 operations = [
41 RunPython(update_results)
42 ]
43
44 # ... other stuff
45class A:
46
47 def __init_subclass__(cls, identifier=None, **kwargs):
48 super().__init_subclass__(**kwargs)
49 if identifier is None:
50 raise ValueError()
51 cls.identifier = identifier
52 Type.objects.update_or_create(
53 identifier=identifier,
54 defaults=dict(name=cls.__name__, desc=cls.__doc__)
55 )
56
57 # ... the rest of A
58
59# The identifier should never change, so that even if the
60# name of the class changes, we still know which subclass is referred to
61class B(A, identifier=3):
62
63 # ... the rest of B
64from django.apps import AppConfig
65from django.db.models.signals import post_migrate
66
67
68def get_all_subclasses(cls):
69 """Get all subclasses of a class, recursively.
70
71 Used to get a list of all the implemented As.
72 """
73 all_subclasses = []
74
75 for subclass in cls.__subclasses__():
76 all_subclasses.append(subclass)
77 all_subclasses.extend(get_all_subclasses(subclass))
78
79 return all_subclasses
80
81
82def update_As(sender=None, **kwargs):
83 """Get a list of all implemented As and write them in the database.
84
85 More precisely, each model is used to instantiate a Type, which will be used to identify As.
86 """
87 from app.models import A, Type
88
89 subclasses = get_all_subclasses(A)
90 for cls in subclasses:
91 id = cls.identifier
92 Type.objects.update_or_create(identifier=id, defaults=dict(name=cls.__name__, desc=cls.__doc__))
93
94
95class MyAppConfig(AppConfig):
96 default_auto_field = "django.db.models.BigAutoField"
97 name = "app"
98
99 def ready(self):
100 post_migrate.connect(update_As, sender=self)
101Hope this is helpful for future Django coders in need!
QUESTION
How to abandon a manual Room migration script, and fall back to destructive migration?
Asked 2022-Mar-04 at 04:22In the app I'm working on, we had a complex manual migration that required data parsing, manual SQL commands, etc. This was to convert a List<X> column into a new linked table of X. I've previously written about the approach, but the specific commands are not especially relevant for this question.
The issue I'm encountering is ~1% of users are experiencing a crash as part of this migration. This cannot be reproduced in testing, and due to our table's size, Crashlytics cannot show any useful error:
Losing customer data isn't catastrophic in this context, but being stuck in the current "try migrate, crash, reopen app and repeat" loop is. As such, I want to just give up on the migration and fall back to a destructive migration if we encounter an exception.
Any ideas how this can be done? My current solution is rerunning the DB changes (but not the presumably failing data migration) inside the catch, but this feels very hacky.
Our database is defined as:
1Room.databaseBuilder(
2 context.applicationContext,
3 CreationDatabase::class.java,
4 "creation_database"
5 )
6 .addMigrations(MIGRATION_11_12, MIGRATION_14_15)
7 .fallbackToDestructiveMigration()
8 .build()
9where MIGRATION_14_15 is:
1Room.databaseBuilder(
2 context.applicationContext,
3 CreationDatabase::class.java,
4 "creation_database"
5 )
6 .addMigrations(MIGRATION_11_12, MIGRATION_14_15)
7 .fallbackToDestructiveMigration()
8 .build()
9private val MIGRATION_14_15 = object : Migration(14, 15) {
10 override fun migrate(database: SupportSQLiteDatabase) {
11 try {
12 // database.execSQL create table etc
13 } catch (e: Exception) {
14 e.printStackTrace()
15 // Here is where I want to give up, and start the DB from scratch
16 }
17 }
18}
19ANSWER
Answered 2022-Mar-04 at 04:22The problem you have is that you cannot (at least easily) invoke the fall-back as that is only invoked when there is no migration.
What you could do is to mimic what fall back does (well close to what it does). That is the fall-back will delete (I think) the database file and create the database from scratch and then invoke the databases _Impl (generated java) createAllTables method.
However, you would likely have issues if you deleted the file as the database connection has been passed to the migration.
So instead you could DROP all the app's tables using the code copied from the dropAllTables method from the generated java. You could then follow this with the code from the createAllTables method.
- These methods are in the generated java as the class that is the same as the class that is annotated with @Database suffixed with _Impl.
The gotcha, is that the exception
(Expected .... Found ....) that you have shown is NOT within the migration but after the migration when Room is trying to build the database, so you have no control/place to do the above fall-back mimic unless this was done for all 14-15 migrations.
Perhaps what you could do is to trap the exception, present a dialog requesting the user to uninstall the app and to then re-install. This would then bypass the migration as it would be a fresh install.
QUESTION
Copy SQL Server Scheduled Jobs to Oracle Scheduler
Asked 2022-Mar-02 at 12:25I have a requirement of migrating from SQL Server to Oracle 19C. Data migration is done and now I want to know whether there is a mechanism in Oracle to copy SQL Server scheduled job to Oracle Scheduler?
ANSWER
Answered 2022-Mar-02 at 12:25Non, there is not. You have to understand what each work do and traduce it in the ways of Oracle do, then schedule the work with Oracle Scheduler.
QUESTION
Using SchemaCrawler API to get Weak/Implied Relationships
Asked 2022-Feb-17 at 23:47I'm using the SchemaCrawler API in a data migration tool I'm developing. This is running on MySQL 5.6.25 with v16.16.9 of the SchemaCrawler API. Unfortunately, SC isn't finding any weak relationships in the database, even though other tools are identifying them. In the sample below, I'm looping through every table in the database and printing the weak associations, but the weak associations collection is always empty:
1for (Map.Entry<String, Table> entry : tableMap.entrySet()) {
2 String tableName = entry.getKey();
3 Table sampleTable = entry.getValue();
4 Collection<WeakAssociation> weakAssociations = sampleTable.getWeakAssociations();
5 System.out.println(String.format("Table: %s, associations=%s", tableName, weakAssociations));
6}
7I couldn't find any sample code relating to weak associations, so maybe I'm just using it incorrectly. Any help is appeciated.
ANSWER
Answered 2022-Feb-17 at 23:47You need to set "weak-associations" to true in the configuration. Please take a look at WeakAssociationsTest for example code.
Sualeh Fatehi, SchemaCrawler
QUESTION
MySQL idempotent version of add column failing
Asked 2022-Feb-14 at 18:45MySQL here. Trying to add a column to a table in idempotent fashion. In reality it will be a SQL script that gets ran as part of an application data migration, so it will be ran over and over and I want to make sure that we only run it if the column does not already exist.
My best attempt so far:
1IF NOT EXISTS (SELECT 1
2 FROM information_schema.COLUMNS
3 WHERE
4 TABLE_SCHEMA = 'myapp'
5 AND TABLE_NAME = 'mytable'
6 AND COLUMN_NAME = 'fizzbuzz')
7BEGIN
8 alter table myapp.mytable
9 add column fizzbuzz tinyint(1) not null default false;
10END
11yields a vague syntax error:
1IF NOT EXISTS (SELECT 1
2 FROM information_schema.COLUMNS
3 WHERE
4 TABLE_SCHEMA = 'myapp'
5 AND TABLE_NAME = 'mytable'
6 AND COLUMN_NAME = 'fizzbuzz')
7BEGIN
8 alter table myapp.mytable
9 add column fizzbuzz tinyint(1) not null default false;
10END
11"IF" is not valid at this position, expecting EOF, ALTER, ANALYZE, BEGIN, BINLOG, CACHE, ...
12Can anyone spot where my syntax is going awry?
ANSWER
Answered 2022-Feb-14 at 18:45use:
1IF NOT EXISTS (SELECT 1
2 FROM information_schema.COLUMNS
3 WHERE
4 TABLE_SCHEMA = 'myapp'
5 AND TABLE_NAME = 'mytable'
6 AND COLUMN_NAME = 'fizzbuzz')
7BEGIN
8 alter table myapp.mytable
9 add column fizzbuzz tinyint(1) not null default false;
10END
11"IF" is not valid at this position, expecting EOF, ALTER, ANALYZE, BEGIN, BINLOG, CACHE, ...
12ALTER TABLE myapp.mytable
13 ADD COLUMN IF NOT exists fizzbuzz TINYINT(1) NOT NULL DEFAULT FALSE;
14Community Discussions contain sources that include Stack Exchange Network
Tutorials and Learning Resources in Data Migration
Tutorials and Learning Resources are not available at this moment for Data Migration
Share this Page
Get latest updates on Data Migration