By continuing you indicate that you have read and agree to our Terms of service and Privacy policy
by tensorflow C++ Version: 1.15.0 License: Apache-2.0
by tensorflow C++ Version: 1.15.0 License: Apache-2.0
Support
Quality
Security
License
Reuse
kandi has reviewed tensorflow and discovered the below as its top functions. This is intended to give you an instant insight into tensorflow implemented functionality, and help decide if they suit your requirements.
Get all kandi verified functions for this library.
Get all kandi verified functions for this library.
An Open Source Machine Learning Framework for Everyone
See all related Code Snippets
QUESTION
What is XlaBuilder for?
Asked 2022-Mar-20 at 18:41What's the XLA class XlaBuilder
for? The docs describe its interface but don't provide a motivation.
The presentation in the docs, and indeed the comment above XlaBuilder
in the source code
// A convenient interface for building up computations.
suggests it's no more than a utility. However, this doesn't appear to explain its behaviour in other places. For example, we can construct an XlaOp
with an XlaBuilder
via e.g.
XlaOp ConstantLiteral(XlaBuilder* builder, const LiteralSlice& literal);
Here, it's not clear to me what role builder
plays (note functions for constructing XlaOp
s aren't documented on the published docs). Further, when I add two XlaOp
s (with +
or Add
) it appears the ops must be constructed with the same builder, else I see
F tensorflow/core/platform/statusor.cc:33] Attempting to fetch value instead of handling error Invalid argument: No XlaOp with handle -1
Indeed, XlaOp
retains a handle for an XlaBuilder
. This suggests to me that the XlaBuilder
has a more fundamental significance.
Beyond the title question, is there a use case for using multiple XlaBuilder
s, or would you typically use one global instance for everything?
ANSWER
Answered 2021-Dec-15 at 01:32XlaBuilder
is the C++ API for building up XLA computations -- conceptually this is like building up a function, full of various operations, that you could execute over and over again on different input data.
Some background, XLA serves as an abstraction layer for creating executable blobs that run on various target accelerators (CPU, GPU, TPU, IPU, ...), conceptually kind of an "accelerator virtual machine" with conceptual similarities to earlier systems like PeakStream or the line of work that led to ArBB.
The XlaBuilder
is a way to enqueue operations into a "computation" (similar to a function) that you want to run against the various set of accelerators that XLA can target. The operations at this level are often referred to as "High Level Operations" (HLOs).
The returned XlaOp
represents the result of the operation you've just enqueued. (Aside/nerdery: this is a classic technique used in "builder" APIs that represent the program in "Static Single Assignment" form under the hood, the operation itself and the result of the operation can be unified as one concept!)
XLA computations are very similar to functions, so you can think of what you're doing with an XlaBuilder
like building up a function. (Aside: they're called "computations" because they do a little bit more than a straightforward function -- conceptually they are coroutines that can talk to an external "host" world and also talk to each other via networking facilities.)
So the fact XlaOp
s can't be used across XlaBuilder
s may make more sense with that context -- in the same way that when building up a function you can't grab intermediate results in the internals of other functions, you have to compose them with function calls / parameters. In XlaBuilder
you can Call
another built computation, which is a reason you might use multiple builders.
As you note, you can choose to inline everything into one "mega builder", but often programs are structured as functions that get composed together, and ultimately get called from a few different "entry points". XLA currently aggressively specializes for the entry points it sees API users using, but this is a design artifact similar to inlining decisions, XLA can conceptually reuse computations built up / invoked from multiple callers if it thought that was the right thing to do. Usually it's most natural to enqueue things into XLA however is convenient for your description from the "outside world", and allow XLA to inline and aggressively specialize the "entry point" computations you've built up as you execute them, in Just-in-Time compilation fashion.
QUESTION
WebSocket not working when trying to send generated answer by keras
Asked 2022-Feb-17 at 12:52I am implementing a simple chatbot using keras and WebSockets. I now have a model that can make a prediction about the user input and send the according answer.
When I do it through command line it works fine, however when I try to send the answer through my WebSocket, the WebSocket doesn't even start anymore.
Here is my working WebSocket code:
@sock.route('/api')
def echo(sock):
while True:
# get user input from browser
user_input = sock.receive()
# print user input on console
print(user_input)
# read answer from console
response = input()
# send response to browser
sock.send(response)
Here is my code to communicate with the keras model on command line:
while True:
question = input("")
ints = predict(question)
answer = response(ints, json_data)
print(answer)
Used methods are those:
def predict(sentence):
bag_of_words = convert_sentence_in_bag_of_words(sentence)
# pass bag as list and get index 0
prediction = model.predict(np.array([bag_of_words]))[0]
ERROR_THRESHOLD = 0.25
accepted_results = [[tag, probability] for tag, probability in enumerate(prediction) if probability > ERROR_THRESHOLD]
accepted_results.sort(key=lambda x: x[1], reverse=True)
output = []
for accepted_result in accepted_results:
output.append({'intent': classes[accepted_result[0]], 'probability': str(accepted_result[1])})
print(output)
return output
def response(intents, json):
tag = intents[0]['intent']
intents_as_list = json['intents']
for i in intents_as_list:
if i['tag'] == tag:
res = random.choice(i['responses'])
break
return res
So when I start the WebSocket with the working code I get this output:
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Serving Flask app 'server' (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
But as soon as I have anything of my model in the server.py
class I get this output:
2022-02-13 11:31:38.887640: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-02-13 11:31:38.887734: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
Metal device set to: Apple M1
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB
It is enough when I just have an import at the top like this: from chatty import response, predict
- even though they are unused.
ANSWER
Answered 2022-Feb-16 at 19:53There is no problem with your websocket route. Could you please share how you are triggering this route? Websocket is a different protocol and I'm suspecting that you are using a HTTP client to test websocket. For example in Postman:
HTTP requests are different than websocket requests. So, you should use appropriate client to test websocket.
QUESTION
Could not resolve com.google.guava:guava:30.1-jre - Gradle project sync failed. Basic functionality will not work properly - in kotlin project
Asked 2022-Feb-14 at 19:47It was a project that used to work well in the past, but after updating, the following errors appear.
plugins {
id 'com.android.application'
id 'kotlin-android'
}
android {
compileSdkVersion 30
buildToolsVersion "30.0.3"
defaultConfig {
applicationId "com.example.retrofit_test"
minSdkVersion 21
targetSdkVersion 30
versionCode 1
versionName "1.0"
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
}
}
compileOptions {
sourceCompatibility JavaVersion.VERSION_1_8
targetCompatibility JavaVersion.VERSION_1_8
}
kotlinOptions {
jvmTarget = '1.8'
}
}
dependencies {
// implementation 'com.google.guava:guava:30.1.1-jre'
implementation 'com.google.guava:guava:30.1-jre'
// implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.5.30-M1'
implementation 'androidx.core:core-ktx:1.6.0'
implementation 'androidx.appcompat:appcompat:1.3.1'
implementation 'com.google.android.material:material:1.4.0'
implementation 'androidx.constraintlayout:constraintlayout:2.1.0'
implementation 'androidx.navigation:navigation-fragment-ktx:2.3.5'
implementation 'androidx.navigation:navigation-ui-ktx:2.3.5'
implementation 'androidx.lifecycle:lifecycle-livedata-ktx:2.3.1'
implementation 'androidx.lifecycle:lifecycle-viewmodel-ktx:2.3.1'
implementation 'androidx.navigation:navigation-fragment-ktx:2.3.5'
implementation 'androidx.navigation:navigation-ui-ktx:2.3.5'
implementation 'com.squareup.retrofit2:retrofit:2.9.0'
implementation 'com.google.code.gson:gson:2.8.7'
implementation 'com.squareup.retrofit2:converter-gson:2.9.0'
implementation 'com.squareup.okhttp3:logging-interceptor:4.9.1'
implementation 'com.github.bumptech.glide:glide:4.12.0'
implementation 'android.arch.persistence.room:guava:1.1.1'
annotationProcessor 'com.github.bumptech.glide:compiler:4.11.0'
testImplementation 'junit:junit:4.13.2'
androidTestImplementation 'androidx.test.ext:junit:1.1.3'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
}
If we need more source code to check, I will update it.
The error contents are as follows.
A problem occurred configuring root project 'Retrofit_Test'.
Could not resolve all artifacts for configuration ':classpath'.
Could not find org.jetbrains.kotlin:kotlin-gradle-plugin:1.5.30.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/jetbrains/kotlin/kotlin-gradle-plugin/1.5.30/kotlin-gradle-plugin-1.5.30.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project :
Could not find org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.4.32.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.4.32/kotlin-stdlib-jdk8-1.4.32.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:repository:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:aaptcompiler:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.analytics-library:shared:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.lint:lint-model:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > androidx.databinding:databinding-compiler-common:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-host-retention-proto:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder-model:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:gradle-api:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2 > com.android.tools:common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.analytics-library:tracker:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.build:manifest-merger:30.0.2
Could not find org.apache.httpcomponents:httpmime:4.5.6.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/apache/httpcomponents/httpmime/4.5.6/httpmime-4.5.6.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdklib:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.analytics-library:crash:30.0.2
Could not find commons-io:commons-io:2.4.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/commons-io/commons-io/2.4/commons-io-2.4.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > androidx.databinding:databinding-compiler-common:7.0.2
Could not find org.ow2.asm:asm:7.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/ow2/asm/asm/7.0/asm-7.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:gradle-api:7.0.2
Could not find org.ow2.asm:asm-analysis:7.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/ow2/asm/asm-analysis/7.0/asm-analysis-7.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
Could not find org.ow2.asm:asm-commons:7.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/ow2/asm/asm-commons/7.0/asm-commons-7.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
Could not find org.ow2.asm:asm-util:7.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/ow2/asm/asm-util/7.0/asm-util-7.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
Could not find org.bouncycastle:bcpkix-jdk15on:1.56.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/bouncycastle/bcpkix-jdk15on/1.56/bcpkix-jdk15on-1.56.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.build:apkzlib:7.0.2
Could not find org.glassfish.jaxb:jaxb-runtime:2.3.2.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/glassfish/jaxb/jaxb-runtime/2.3.2/jaxb-runtime-2.3.2.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdklib:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:repository:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > androidx.databinding:databinding-compiler-common:7.0.2
Could not find net.sf.jopt-simple:jopt-simple:4.9.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/net/sf/jopt-simple/jopt-simple/4.9/jopt-simple-4.9.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
Could not find com.squareup:javapoet:1.10.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/com/squareup/javapoet/1.10.0/javapoet-1.10.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > androidx.databinding:databinding-compiler-common:7.0.2
Could not find com.google.protobuf:protobuf-java:3.10.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/com/google/protobuf/protobuf-java/3.10.0/protobuf-java-3.10.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.ddms:ddmlib:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:aapt2-proto:7.0.2-7396180
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:aaptcompiler:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-device-provider-gradle-proto:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-host-retention-proto:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-result-listener-gradle-proto:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:bundletool:1.6.0
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.analytics-library:shared:30.0.2 > com.android.tools.analytics-library:protos:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.analytics-library:tracker:30.0.2
Could not find com.google.protobuf:protobuf-java-util:3.10.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/com/google/protobuf/protobuf-java-util/3.10.0/protobuf-java-util-3.10.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:bundletool:1.6.0
Could not find com.google.code.gson:gson:2.8.6.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/com/google/code/gson/gson/2.8.6/gson-2.8.6.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdklib:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.analytics-library:shared:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > androidx.databinding:databinding-compiler-common:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-result-listener-gradle-proto:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.build:manifest-merger:30.0.2
Could not find io.grpc:grpc-core:1.21.1.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/io/grpc/grpc-core/1.21.1/grpc-core-1.21.1.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-result-listener-gradle-proto:30.0.2
Could not find io.grpc:grpc-netty:1.21.1.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/io/grpc/grpc-netty/1.21.1/grpc-netty-1.21.1.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-result-listener-gradle-proto:30.0.2
Could not find io.grpc:grpc-protobuf:1.21.1.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/io/grpc/grpc-protobuf/1.21.1/grpc-protobuf-1.21.1.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-result-listener-gradle-proto:30.0.2
Could not find io.grpc:grpc-stub:1.21.1.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/io/grpc/grpc-stub/1.21.1/grpc-stub-1.21.1.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-result-listener-gradle-proto:30.0.2
Could not find com.google.crypto.tink:tink:1.3.0-rc2.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/com/google/crypto/tink/tink/1.3.0-rc2/tink-1.3.0-rc2.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
Could not find com.google.flatbuffers:flatbuffers-java:1.12.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/com/google/flatbuffers/flatbuffers-java/1.12.0/flatbuffers-java-1.12.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
Could not find org.tensorflow:tensorflow-lite-metadata:0.1.0-rc2.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/tensorflow/tensorflow-lite-metadata/0.1.0-rc2/tensorflow-lite-metadata-0.1.0-rc2.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2
Could not find org.bouncycastle:bcprov-jdk15on:1.56.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.build:apkzlib:7.0.2
Could not find com.google.guava:guava:30.1-jre.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:aaptcompiler:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.analytics-library:crash:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.analytics-library:shared:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > androidx.databinding:databinding-compiler-common:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder-test-api:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.utp:android-test-plugin-result-listener-gradle-proto:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:bundletool:1.6.0
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:gradle-api:7.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2 > com.android.tools:common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.analytics-library:tracker:30.0.2
Could not find org.jetbrains.kotlin:kotlin-reflect:1.4.32.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/jetbrains/kotlin/kotlin-reflect/1.4.32/kotlin-reflect-1.4.32.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
Could not find javax.inject:javax.inject:1.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/javax/inject/javax.inject/1/javax.inject-1.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:bundletool:1.6.0
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2
Could not find net.sf.kxml:kxml2:2.3.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/net/sf/kxml/kxml2/2.3.0/kxml2-2.3.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.ddms:ddmlib:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.lint:lint-model:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.layoutlib:layoutlib-api:30.0.2
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools.build:builder:7.0.2 > com.android.tools.build:manifest-merger:30.0.2
Could not find org.jetbrains.intellij.deps:trove4j:1.0.20181211.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/jetbrains/intellij/deps/trove4j/1.0.20181211/trove4j-1.0.20181211.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
Could not find xerces:xercesImpl:2.12.0.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/xerces/xercesImpl/2.12.0/xercesImpl-2.12.0.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
Required by:
project : > com.android.tools.build:gradle:7.0.2 > com.android.tools:sdk-common:30.0.2
Could not find org.apache.commons:commons-compress:1.20.
Searched in the following locations:
- https://dl.google.com/dl/android/maven2/org/apache/commons/commons-compress/1.20/commons-compress-1.20.pom
If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
ANSWER
Answered 2021-Sep-17 at 11:03Add mavenCentral() in Build Script
repositories {
mavenCentral()
google()
}
QUESTION
Tensorflow setup on RStudio/ R | CentOS
Asked 2022-Feb-11 at 09:36For the last 5 days, I am trying to make Keras/Tensorflow packages work in R. I am using RStudio for installation and have used conda
, miniconda
, virtualenv
but it crashes each time in the end. Installing a library should not be a nightmare especially when we are talking about R (one of the best statistical languages) and TensorFlow (one of the best deep learning libraries). Can someone share a reliable way to install Keras/Tensorflow on CentOS 7?
Following are the steps I am using to install tensorflow
in RStudio.
Since RStudio simply crashes each time I run tensorflow::tf_config()
I have no way to check what is going wrong.
devtools::install_github("rstudio/reticulate")
devtools::install_github("rstudio/keras") # This package also installs tensorflow
library(reticulate)
reticulate::install_miniconda()
reticulate::use_miniconda("r-reticulate")
library(tensorflow)
tensorflow::tf_config() **# Crashes at this point**
sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] tensorflow_2.7.0.9000 keras_2.7.0.9000 reticulate_1.22-9000
loaded via a namespace (and not attached):
[1] Rcpp_1.0.7 lattice_0.20-45 png_0.1-7 zeallot_0.1.0
[5] rappdirs_0.3.3 grid_3.6.0 R6_2.5.1 jsonlite_1.7.2
[9] magrittr_2.0.1 tfruns_1.5.0 rlang_0.4.12 whisker_0.4
[13] Matrix_1.3-4 generics_0.1.1 tools_3.6.0 compiler_3.6.0
[17] base64enc_0.1-3
Update 1 The only way RStudio does not crash while installing tensorflow is by executing following steps -
First, I created a new virtual environment using conda
conda create --name py38 python=3.8.0
conda activate py38
conda install tensorflow=2.4
Then from within RStudio, I installed reticulate and activated the virtual environment which I earlier created using conda
devtools::install_github("rstudio/reticulate")
library(reticulate)
reticulate::use_condaenv("/root/.conda/envs/py38", required = TRUE)
reticulate::use_python("/root/.conda/envs/py38/bin/python3.8", required = TRUE)
reticulate::py_available(initialize = TRUE)
ts <- reticulate::import("tensorflow")
As soon as I try to import tensorflow
in RStudio, it loads the library /lib64/libstdc++.so.6
instead of /root/.conda/envs/py38/lib/libstdc++.so.6
and I get the following error -
Error in py_module_import(module, convert = convert) :
ImportError: Traceback (most recent call last):
File "/root/.conda/envs/py38/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
from tensorflow.python._pywrap_tensorflow_internal import *
File "/home/R/x86_64-redhat-linux-gnu-library/3.6/reticulate/python/rpytools/loader.py", line 39, in _import_hook
module = _import(
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /root/.conda/envs/py38/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Here is what inside /lib64/libstdc++.so.6
> strings /lib64/libstdc++.so.6 | grep GLIBC
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBC_2.3
GLIBC_2.2.5
GLIBC_2.14
GLIBC_2.4
GLIBC_2.3.2
GLIBCXX_DEBUG_MESSAGE_LENGTH
To resolve the library issue, I added the path of the correct libstdc++.so.6
library having GLIBCXX_3.4.20
in RStudio.
system('export LD_LIBRARY_PATH=/root/.conda/envs/py38/lib/:$LD_LIBRARY_PATH')
and, also
Sys.setenv("LD_LIBRARY_PATH" = "/root/.conda/envs/py38/lib")
But still I get the same error ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20'
. Somehow RStudio still loads /lib64/libstdc++.so.6
first instead of /root/.conda/envs/py38/lib/libstdc++.so.6
Instead of RStudio
, if I execute the above steps in the R
console, then also I get the exact same error.
Update 2: A solution is posted here
ANSWER
Answered 2022-Jan-16 at 00:08Perhaps my failed attempts will help someone else solve this problem; my approach:
sudo yum install epel-release
sudo yum install R
sudo yum install libxml2-devel
sudo yum install openssl-devel
sudo yum install libcurl-devel
sudo yum install libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver
conda init
conda create --name tf
conda activate tf
conda install -c conda-forge tensorflow
**From within this conda env you can import tensorflow in python without error; now to access tf via R
sudo yum install centos-release-scl
sudo yum install devtoolset-7-gcc*
scl enable devtoolset-7 R
install.packages("remotes")
remotes::install_github('rstudio/reticulate')
reticulate::use_condaenv("tf", conda = "~/anaconda3/bin/conda")
reticulate::repl_python()
# This works as expected but the command "import tensorflow" crashes R
# Error: *** caught segfault *** address 0xf8, cause 'memory not mapped'
# Also tried:
install.packages("devtools")
devtools::install_github('rstudio/tensorflow')
devtools::install_github('rstudio/keras')
library(tensorflow)
install_tensorflow() # "successful"
tensorflow::tf_config()
# Error: *** caught segfault *** address 0xf8, cause 'memory not mapped'
devtools::install_github('rstudio/tensorflow@v2.4.0')
devtools::install_github('rstudio/keras@v2.4.0')
library(tensorflow)
tf_config()
# Error: *** caught segfault *** address 0xf8, cause 'memory not mapped'
# deactivate conda
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
export R_VERSION=4.0.0
curl -O https://cdn.rstudio.com/r/centos-7/pkgs/R-${R_VERSION}-1-1.x86_64.rpm
sudo yum install R-${R_VERSION}-1-1.x86_64.rpm
scl enable devtoolset-7 /opt/R/4.0.0/bin/R
install.packages("devtools")
devtools::install_github('rstudio/reticulate')
reticulate::use_condaenv("tf", conda = "~/anaconda3/bin/conda")
reticulate::repl_python()
# 'import tensorflow' resulted in "core dumped"
I guess the issue is with R/CentOS, as you can import and use tensorflow via python normally, but I'm not sure what else to try.
I would also like to say that I had no issues with Ubuntu (which is specifically supported by tensorflow, along with macOS and Windows), and I came across these docs that might be some help: https://wiki.hpcc.msu.edu/display/ITH/Installing+TensorFlow+using+anaconda / https://wiki.hpcc.msu.edu/pages/viewpage.action?pageId=22709999
QUESTION
Saving model on Tensorflow 2.7.0 with data augmentation layer
Asked 2022-Feb-04 at 17:25I am getting an error when trying to save a model with data augmentation layers with Tensorflow version 2.7.0.
Here is the code of data augmentation:
input_shape_rgb = (img_height, img_width, 3)
data_augmentation_rgb = tf.keras.Sequential(
[
layers.RandomFlip("horizontal"),
layers.RandomFlip("vertical"),
layers.RandomRotation(0.5),
layers.RandomZoom(0.5),
layers.RandomContrast(0.5),
RandomColorDistortion(name='random_contrast_brightness/none'),
]
)
Now I build my model like this:
# Build the model
input_shape = (img_height, img_width, 3)
model = Sequential([
layers.Input(input_shape),
data_augmentation_rgb,
layers.Rescaling((1./255)),
layers.Conv2D(16, kernel_size, padding=padding, activation='relu', strides=1,
data_format='channels_last'),
layers.MaxPooling2D(),
layers.BatchNormalization(),
layers.Conv2D(32, kernel_size, padding=padding, activation='relu'), # best 4
layers.MaxPooling2D(),
layers.BatchNormalization(),
layers.Conv2D(64, kernel_size, padding=padding, activation='relu'), # best 3
layers.MaxPooling2D(),
layers.BatchNormalization(),
layers.Conv2D(128, kernel_size, padding=padding, activation='relu'), # best 3
layers.MaxPooling2D(),
layers.BatchNormalization(),
layers.Flatten(),
layers.Dense(128, activation='relu'), # best 1
layers.Dropout(0.1),
layers.Dense(128, activation='relu'), # best 1
layers.Dropout(0.1),
layers.Dense(64, activation='relu'), # best 1
layers.Dropout(0.1),
layers.Dense(num_classes, activation = 'softmax')
])
model.compile(loss='categorical_crossentropy', optimizer='adam',metrics=metrics)
model.summary()
Then after the training is done I just make:
model.save("./")
And I'm getting this error:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-84-87d3f09f8bee> in <module>()
----> 1 model.save("./")
/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in
error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
/usr/local/lib/python3.7/dist-
packages/tensorflow/python/saved_model/function_serialization.py in
serialize_concrete_function(concrete_function, node_ids, coder)
66 except KeyError:
67 raise KeyError(
---> 68 f"Failed to add concrete function '{concrete_function.name}' to
object-"
69 f"based SavedModel as it captures tensor {capture!r} which is
unsupported"
70 " or not reachable from root. "
KeyError: "Failed to add concrete function
'b'__inference_sequential_46_layer_call_fn_662953'' to object-based SavedModel as it
captures tensor <tf.Tensor: shape=(), dtype=resource, value=<Resource Tensor>> which
is unsupported or not reachable from root. One reason could be that a stateful
object or a variable that the function depends on is not assigned to an attribute of
the serialized trackable object (see SaveTest.test_captures_unreachable_variable)."
I inspected the reason of getting this error by changing the architecture of my model and I just found that reason came from the data_augmentation layer since the RandomFlip
and RandomRotation
and others are changed from layers.experimental.prepocessing.RandomFlip
to layers.RandomFlip
, but still the error appears.
ANSWER
Answered 2022-Feb-04 at 17:25This seems to be a bug in Tensorflow 2.7 when using model.save
combined with the parameter save_format="tf"
, which is set by default. The layers RandomFlip
, RandomRotation
, RandomZoom
, and RandomContrast
are causing the problems, since they are not serializable. Interestingly, the Rescaling
layer can be saved without any problems. A workaround would be to simply save your model with the older Keras H5 format model.save("test", save_format='h5')
:
import tensorflow as tf
import numpy as np
class RandomColorDistortion(tf.keras.layers.Layer):
def __init__(self, contrast_range=[0.5, 1.5],
brightness_delta=[-0.2, 0.2], **kwargs):
super(RandomColorDistortion, self).__init__(**kwargs)
self.contrast_range = contrast_range
self.brightness_delta = brightness_delta
def call(self, images, training=None):
if not training:
return images
contrast = np.random.uniform(
self.contrast_range[0], self.contrast_range[1])
brightness = np.random.uniform(
self.brightness_delta[0], self.brightness_delta[1])
images = tf.image.adjust_contrast(images, contrast)
images = tf.image.adjust_brightness(images, brightness)
images = tf.clip_by_value(images, 0, 1)
return images
def get_config(self):
config = super(RandomColorDistortion, self).get_config()
config.update({"contrast_range": self.contrast_range, "brightness_delta": self.brightness_delta})
return config
input_shape_rgb = (256, 256, 3)
data_augmentation_rgb = tf.keras.Sequential(
[
tf.keras.layers.RandomFlip("horizontal"),
tf.keras.layers.RandomFlip("vertical"),
tf.keras.layers.RandomRotation(0.5),
tf.keras.layers.RandomZoom(0.5),
tf.keras.layers.RandomContrast(0.5),
RandomColorDistortion(name='random_contrast_brightness/none'),
]
)
input_shape = (256, 256, 3)
padding = 'same'
kernel_size = 3
model = tf.keras.Sequential([
tf.keras.layers.Input(input_shape),
data_augmentation_rgb,
tf.keras.layers.Rescaling((1./255)),
tf.keras.layers.Conv2D(16, kernel_size, padding=padding, activation='relu', strides=1,
data_format='channels_last'),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Conv2D(32, kernel_size, padding=padding, activation='relu'), # best 4
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Conv2D(64, kernel_size, padding=padding, activation='relu'), # best 3
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Conv2D(128, kernel_size, padding=padding, activation='relu'), # best 3
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'), # best 1
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(128, activation='relu'), # best 1
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(64, activation='relu'), # best 1
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(5, activation = 'softmax')
])
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()
model.save("test", save_format='h5')
Loading your model with your custom layer would look like this then:
model = tf.keras.models.load_model('test.h5', custom_objects={'RandomColorDistortion': RandomColorDistortion})
where RandomColorDistortion
is the name of your custom layer.
QUESTION
Is it possible to use a collection of hyperspectral 1x1 pixels in a CNN model purposed for more conventional datasets (CIFAR-10/MNIST)?
Asked 2021-Dec-17 at 09:08I have created a working CNN model in Keras/Tensorflow, and have successfully used the CIFAR-10 & MNIST datasets to test this model. The functioning code as seen below:
import keras
from keras.datasets import cifar10
from keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Conv2D, Flatten, MaxPooling2D
from keras.layers.normalization import BatchNormalization
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
#reshape data to fit model
X_train = X_train.reshape(50000,32,32,3)
X_test = X_test.reshape(10000,32,32,3)
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
# Building the model
#1st Convolutional Layer
model.add(Conv2D(filters=64, input_shape=(32,32,3), kernel_size=(11,11), strides=(4,4), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))
#2nd Convolutional Layer
model.add(Conv2D(filters=224, kernel_size=(5, 5), strides=(1,1), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))
#3rd Convolutional Layer
model.add(Conv2D(filters=288, kernel_size=(3,3), strides=(1,1), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
#4th Convolutional Layer
model.add(Conv2D(filters=288, kernel_size=(3,3), strides=(1,1), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
#5th Convolutional Layer
model.add(Conv2D(filters=160, kernel_size=(3,3), strides=(1,1), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))
model.add(Flatten())
# 1st Fully Connected Layer
model.add(Dense(4096, input_shape=(32,32,3,)))
model.add(BatchNormalization())
model.add(Activation('relu'))
# Add Dropout to prevent overfitting
model.add(Dropout(0.4))
#2nd Fully Connected Layer
model.add(Dense(4096))
model.add(BatchNormalization())
model.add(Activation('relu'))
#Add Dropout
model.add(Dropout(0.4))
#3rd Fully Connected Layer
model.add(Dense(1000))
model.add(BatchNormalization())
model.add(Activation('relu'))
#Add Dropout
model.add(Dropout(0.4))
#Output Layer
model.add(Dense(10))
model.add(BatchNormalization())
model.add(Activation('softmax'))
#compile model using accuracy to measure model performance
opt = keras.optimizers.Adam(learning_rate = 0.0001)
model.compile(optimizer=opt, loss='categorical_crossentropy',
metrics=['accuracy'])
#train the model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=30)
From this point after utilising the aforementioned datasets, I wanted to go one further and use a dataset with more channels than a greyscale or rgb presented, hence the inclusion of a hyperspectral dataset. When looking for a hyperspectral dataset I came across this one.
The issue at this stage was realising that this hyperspectral dataset was one image, with each value in the ground truth relating to each pixel. At this stage I reformatted the data from this into a collection of hyperspectral data/pixels.
Code reformatting corrected dataset for x_train & x_test:
import keras
import scipy
import numpy as np
import matplotlib.pyplot as plt
from keras.utils import to_categorical
from scipy import io
mydict = scipy.io.loadmat('Indian_pines_corrected.mat')
dataset = np.array(mydict.get('indian_pines_corrected'))
#This is creating the split between x_train and x_test from the original dataset
# x_train after this code runs will have a shape of (121, 145, 200)
# x_test after this code runs will have a shape of (24, 145, 200)
x_train = np.zeros((121,145,200), dtype=np.int)
x_test = np.zeros((24,145,200), dtype=np.int)
xtemp = np.array_split(dataset, [121])
x_train = np.array(xtemp[0])
x_test = np.array(xtemp[1])
# x_train will have a shape of (17545, 200)
# x_test will have a shape of (3480, 200)
x_train = x_train.reshape(-1, x_train.shape[-1])
x_test = x_test.reshape(-1, x_test.shape[-1])
Code reformatting ground truth dataset for Y_train & Y_test:
truthDataset = scipy.io.loadmat('Indian_pines_gt.mat')
gTruth = truthDataset.get('indian_pines_gt')
#This is creating the split between Y_train and Y_test from the original dataset
# Y_train after this code runs will have a shape of (121, 145)
# Y_test after this code runs will have a shape of (24, 145)
Y_train = np.zeros((121,145), dtype=np.int)
Y_test = np.zeros((24,145), dtype=np.int)
ytemp = np.array_split(gTruth, [121])
Y_train = np.array(ytemp[0])
Y_test = np.array(ytemp[1])
# Y_train will have a shape of (17545)
# Y_test will have a shape of (3480)
Y_train = Y_train.reshape(-1)
Y_test = Y_test.reshape(-1)
#17 binary categories ranging from 0-16
#Y_train one-hot encode target column
Y_train = to_categorical(Y_train)
#Y_test one-hot encode target column
Y_test = to_categorical(Y_test, num_classes = 17)
My thought process was that, despite the initial image being broken down into 1x1 patches, the large number of channels each patch possessed with their respective values would aid in categorisation of the dataset.
Essentially I'd want to input this reformatted data into my model (seen within the first code fragment in this post), however I'm uncertain if I am taking the wrong approach to this due to my inexperience with this area of expertise. I was expecting to input a shape of (1,1,200), i.e the shape of x_train & x_test would be (17545,1,1,200) & (3480,1,1,200) respectively.
ANSWER
Answered 2021-Dec-16 at 10:18If the hyperspectral dataset is given to you as a large image with many channels, I suppose that the classification of each pixel should depend on the pixels around it (otherwise I would not format the data as an image, i.e. without grid structure). Given this assumption, breaking up the input picture into 1x1 parts is not a good idea as you are loosing the grid structure.
I further suppose that the order of the channels is arbitrary, which implies that convolution over the channels is probably not meaningful (which you however did not plan to do anyways).
Instead of reformatting the data the way you did, you may want to create a model that takes an image as input and also outputs an "image" containing the classifications for each pixel. I.e. if you have 10 classes and take a (145, 145, 200) image as input, your model would output a (145, 145, 10) image. In that architecture you would not have any fully-connected layers. Your output layer would also be a convolutional layer.
That however means that you will not be able to keep your current architecture. That is because the tasks for MNIST/CIFAR10 and your hyperspectral dataset are not the same. For MNIST/CIFAR10 you want to classify an image in it's entirety, while for the other dataset you want to assign a class to each pixel (while most likely also using the pixels around each pixel).
Some further ideas:
QUESTION
ImportError: cannot import name 'BatchNormalization' from 'keras.layers.normalization'
Asked 2021-Nov-13 at 07:14i have an import problem when executing my code:
from keras.models import Sequential
from keras.layers.normalization import BatchNormalization
2021-10-06 22:27:14.064885: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-10-06 22:27:14.064974: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
File "C:\Data\breast-cancer-classification\train_model.py", line 10, in <module>
from cancernet.cancernet import CancerNet
File "C:\Data\breast-cancer-classification\cancernet\cancernet.py", line 2, in <module>
from keras.layers.normalization import BatchNormalization
ImportError: cannot import name 'BatchNormalization' from 'keras.layers.normalization' (C:\Users\Catalin\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\layers\normalization\__init__.py)
The library it is installed also with
pip install numpy opencv-python pillow tensorflow keras imutils scikit-learn matplotlib
Do you have any ideas?
ANSWER
Answered 2021-Oct-06 at 20:27You're using outdated imports for tf.keras
. Layers can now be imported directly from tensorflow.keras.layers
:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (
BatchNormalization, SeparableConv2D, MaxPooling2D, Activation, Flatten, Dropout, Dense
)
from tensorflow.keras import backend as K
class CancerNet:
@staticmethod
def build(width, height, depth, classes):
model = Sequential()
shape = (height, width, depth)
channelDim = -1
if K.image_data_format() == "channels_first":
shape = (depth, height, width)
channelDim = 1
model.add(SeparableConv2D(32, (3, 3), padding="same", input_shape=shape))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=channelDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(SeparableConv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=channelDim))
model.add(SeparableConv2D(64, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=channelDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(SeparableConv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=channelDim))
model.add(SeparableConv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=channelDim))
model.add(SeparableConv2D(128, (3, 3), padding="same"))
model.add(Activation("relu"))
model.add(BatchNormalization(axis=channelDim))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(256))
model.add(Activation("relu"))
model.add(BatchNormalization())
model.add(Dropout(0.5))
model.add(Dense(classes))
model.add(Activation("softmax"))
return model
model = CancerNet()
QUESTION
Accuracy in Calculating Fourth Derivative using Finite Differences in Tensorflow
Asked 2021-Sep-16 at 13:01I am writing a small code to calculate the fourth derivative using the method of finite differences in tensorflow. This is as follows:
def action(y,x):
#spacing between points.
h = (x[-1] - x[0]) / (int(x.shape[0]) - 1)
#fourth derivative
dy4 = (y[4:] - 4*y[3:-1] + 6*y[2:-2] - 4*y[1:-3] + y[:-4])/(h*h*h*h)
return dy4
x = tf.linspace(0.0, 30, 1000)
y = tf.tanh(x)
dy4 = action(y,x)
sess = tf.compat.v1.Session()
plt.plot(sess.run(dy4))
This results in the following graph:
However if I use essentially the same code but just using numpy, the results are much cleaner:
def fourth_deriv(y, x):
h = (x[-1] - x[0]) / (int(x.shape[0]) - 1)
dy = (y[4:] - 4*y[3:-1] + 6*y[2:-2] - 4*y[1:-3] + y[:-4])/(h*h*h*h)
return dy
x = np.linspace(0.0, 30, 1000)
test = fourth_deriv(np.tanh(x), x)
plt.plot(test)
Which gives:
What is the issue here? I was thinking at first that the separation between points could be too small to give an accurate computation, but clearly, that's not the case if numpy can handle it fine.
ANSWER
Answered 2021-Sep-16 at 13:01The issue is related to the choice of floating-point types.
tf.linspace
automatically selects tf.float32
as its type, whilenp.linspace
creates a float64
array, which has much more precision.Making the following modification:
start = tf.constant(0.0, dtype = tf.float64)
end = tf.constant(30.0, dtype = tf.float64)
x = tf.linspace(start, end, 1000)
causes a smooth plot to appear:
It's worth noting further that Tensorflow does include an automatic differentiation, which is crucial for machine learning training and is hence well-tested - you can use gradient tapes to access it and evaluate a fourth derivative without the imprecision of numeric differentiation using finite differences:
with tf.compat.v1.Session() as sess2:
x = tf.Variable(tf.linspace(0, 30, 1000))
sess2.run(tf.compat.v1.initialize_all_variables())
with tf.GradientTape() as t4:
with tf.GradientTape() as t3:
with tf.GradientTape() as t2:
with tf.GradientTape() as t1:
y = tf.tanh(x)
der1 = t1.gradient(y, x)
der2 = t2.gradient(der1, x)
der3 = t3.gradient(der2, x)
der4 = t4.gradient(der3, x)
print(der4)
plt.plot(sess2.run(der4))
The accuracy of this method is far better than can be achieved using finite difference methods. The following code compares the accuracy of auto diff with the accuracy of the finite difference method:
x = np.linspace(0.0, 30, 1000)
sech = 1/np.cosh(x)
theoretical = 16*np.tanh(x) * np.power(sech, 4) - 8*np.power(np.tanh(x), 3)*np.power(sech,2)
finite_diff_err = theoretical[2:-2] - from_finite_diff
autodiff_err = theoretical[2:-2] - from_autodiff[2:-2]
print('Max err with autodiff: %s' % np.max(np.abs(autodiff_err)))
print('Max err with finite difference: %s' % np.max(np.abs(finite_diff_err)))
line, = plt.plot(np.log10(np.abs(autodiff_err)))
line.set_label('Autodiff log error')
line2, = plt.plot(np.log10(np.abs(finite_diff_err)))
line2.set_label('Finite difference log error')
plt.legend()
and yields the following output:
Max err with autodiff: 3.1086244689504383e-15
Max err with a finite difference: 0.007830900165363808
QUESTION
AssertionError: Tried to export a function which references untracked resource
Asked 2021-Sep-07 at 11:23I wrote a unit-test in order to safe a model after noticing that I am not able to do so (anymore) during training.
@pytest.mark.usefixtures("maybe_run_functions_eagerly")
def test_save_model(speech_model: Tuple[TransducerBase, SpeechFeaturesConfig]):
model, speech_features_config = speech_model
speech_features_config: SpeechFeaturesConfig
channels = 3 if speech_features_config.add_delta_deltas else 1
num_mel_bins = speech_features_config.num_mel_bins
enc_inputs = np.random.rand(1, 50, num_mel_bins, channels)
dec_inputs = np.expand_dims(np.random.randint(0, 25, size=10), axis=1)
inputs = enc_inputs, dec_inputs
model(inputs)
# Throws KeyError:
# graph = tf.compat.v1.get_default_graph()
# tensor = graph.get_tensor_by_name("77040:0")
directory = tempfile.mkdtemp(prefix=f"{model.__class__.__name__}_")
try:
model.save(directory)
finally:
shutil.rmtree(directory)
Trying to save the model will always throw the following error:
E AssertionError: Tried to export a function which references untracked resource Tensor("77040:0", shape=(), dtype=resource). TensorFlow objects (e.g. tf.Variable) captured by functions must be tracked by assigning them to an attribute of a tracked object or assigned to an attribute of the main object directly.
E
E Trackable Python objects referring to this tensor (from gc.get_referrers, limited to two hops):
E <tf.Variable 'transformer_transducer/transducer_encoder/inputs_embedding/convolution_stack/conv2d/kernel:0' shape=(3, 3, 3, 32) dtype=float32>
Note: As you can see in the code above, but I am not able to retrieve this tensor with
tf.compat.v1.get_default_graph().get_tensor_by_name("77040:0")
.I tried the following too, but the result is always empty:
model(batch) # Build the model
tensor_name = "77040"
var_names = [var.name for var in model.trainable_weights]
weights = list(filter(lambda var: tensor_name in var, var_names))
var_names = [var.name for var in model.trainable_variables]
variables = list(filter(lambda var: tensor_name in var, var_names))
print(weights)
print(variables)
The problem is that I do not understand why I am getting this because the affected layer is tracked by Keras as you can see in the screenshot below. I took it during a debug-session in the call()
function.
I have no explanation for this and I am running out of ideas what the issue might be here.
The transformations
list in the screenshot is a property of and getting constructed by a layer InputsEmbedding
like so:
class InputsEmbedding(layers.Layer, TimeReduction):
def __init__(self, config: InputsEmbeddingConfig, **kwargs):
super().__init__(**kwargs)
if config.transformations is None or not len(config.transformations):
raise RuntimeError("No transformations provided.")
self.config = config
self.transformations = list()
for transformation in self.config.transformations:
layer_name, layer_params = list(transformation.items())[0]
layer = _get_layer(layer_name, layer_params)
self.transformations.append(layer)
self.init_time_reduction_layer()
def get_config(self):
return self.config.dict()
def _get_layer(name: str, params: dict) -> layers.Layer:
if name == "conv2d_stack":
return ConvolutionStack(**params)
elif name == "stack_frames":
return StackFrames(**params)
else:
raise RuntimeError(f"Unsupported or unknown time-reduction layer {name}")
In order to verify that the problem is not the InputsEmbedding
, I created a unit-text for saving a model that is using just this particular layer.
@pytest.mark.usefixtures("maybe_run_functions_eagerly")
def test_inputs_embedding_save_model():
convolutions = [
"filters=2, kernel_size=(3, 3), strides=(2, 1)",
"filters=4, kernel_size=(3, 3), strides=(2, 1)",
"filters=8, kernel_size=(3, 4), strides=(1, 1)",
]
config = InputsEmbeddingConfig()
config.transformations = [dict(conv2d_stack=dict(convolutions=convolutions)), dict(stack_frames=dict(n=2))]
num_features = 8
num_channels = 3
inputs = layers.Input(shape=(None, num_features, num_channels))
x = inputs
x, _ = InputsEmbedding(config)(x)
model = keras.Model(inputs=inputs, outputs=x)
model.build(input_shape=(1, 20, num_features, num_channels))
directory = tempfile.mkdtemp(prefix=f"{model.__class__.__name__}_")
try:
model.save(directory)
finally:
shutil.rmtree(directory)
Here I am able to save this layer without any issues:
ConvolutionStack
As it seems to be relevant, here is the (rather ugly) implementation of ConvolutionStack
:
from typing import List
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.python.keras.layers import convolutional
from speech.lab.layers import InputsRequirements
from speech.lab.models import conv_util, models_util
class ConvolutionStack(layers.Layer):
def __init__(
self,
convolutions: List[str],
kernel_regularizer: dict = None,
bias_regularizer: dict = None,
**kwargs
):
super().__init__(**kwargs)
self.config = dict(
convolutions=convolutions,
kernel_regularizer=kernel_regularizer,
bias_regularizer=bias_regularizer
)
self.conv_stack_config = [eval(f"dict({convolution})") for convolution in convolutions]
self.conv_blocks = list()
if kernel_regularizer is not None:
kernel_regularizer = models_util.maybe_to_regularizer(kernel_regularizer)
if bias_regularizer is not None:
bias_regularizer = models_util.maybe_to_regularizer(bias_regularizer)
for block_config in self.conv_stack_config:
block = _new_convolution_block(
**block_config,
kernel_regularizer=kernel_regularizer,
bias_regularizer=bias_regularizer,
)
self.conv_blocks.append(block)
self.drop_dim2 = layers.Lambda(tf.squeeze, arguments=dict(axis=-2))
self.expand_last = layers.Lambda(tf.expand_dims, arguments=dict(axis=-1))
@property
def inputs_requirements(self) -> InputsRequirements:
requirements, frame_look_back = conv_util.get_conv2d_stack_requirements(self.conv_stack_config)
first = requirements[0]
t_min, f_size = first["min_size"]
t_grow, f_grow = first["grow_size"]
return InputsRequirements(
frame_look_back=frame_look_back,
t_min=t_min,
t_grow=t_grow,
f_min=f_size,
f_grow=f_grow,
)
def call(self, inputs, training=None, mask=None, **kwargs):
"""
:param inputs:
Tensor taking the form [batch, time, freq, channel]
:param training:
:param mask:
:param kwargs:
:return:
Tensor taking the form [batch, time, freq, 1]
"""
if training:
t_min = self.inputs_requirements.t_min
t_grow = self.inputs_requirements.t_grow
pad = conv_util.get_padding_for_loss(tf.shape(inputs)[1], t_min=t_min, t_grow=t_grow)
inputs = tf.pad(inputs, ((0, 0), (0, pad), (0, 0), (0, 0)))
if mask is not None:
mask = tf.pad(mask, ((0, 0), (0, pad)))
f_min = self.inputs_requirements.f_min
f_grow = self.inputs_requirements.f_grow
assert (inputs.shape[2] - f_min) % f_grow == 0, (
f'Inputs dimension "freq" ' f"expected to be {f_min} + n * {f_grow} but got {inputs.shape[2]} instead."
)
x = inputs
for block in self.conv_blocks:
for layer in block:
if mask is not None and isinstance(layer, convolutional.Conv):
st, _ = layer.strides
kt = tf.maximum(layer.kernel_size[0] - 1, 1)
mask = mask[:, :-kt][:, ::st]
mask = tf.pad(mask, ((0, 0), (0, tf.maximum(2 - layer.kernel_size[0], 0))))
x = layer(x, training=training)
return self.expand_last(self.drop_dim2(x)), mask
def get_config(self):
return self.config
def _new_convolution_block(
filters: int,
kernel_size: tuple,
strides: tuple,
use_bias: bool = False,
use_norm: bool = True,
kernel_regularizer=None,
bias_regularizer=None,
activation=None,
):
assert strides[0] % 2 == 0 or strides[0] == 1, "Strides on the time axis must be divisible by 2 or be exactly 1."
if activation is not None:
activation_layer = layers.Activation(activation)
else:
activation_layer = layers.Lambda(lambda x: x)
if use_norm:
norm_layer = layers.LayerNormalization()
else:
norm_layer = layers.Lambda(lambda x: x)
return (
layers.Conv2D(
filters=filters,
kernel_size=kernel_size,
strides=strides,
use_bias=use_bias,
kernel_regularizer=kernel_regularizer,
bias_regularizer=bias_regularizer,
),
norm_layer,
activation_layer,
)
ANSWER
Answered 2021-Sep-06 at 13:25Your issue is not related to 'transformer_transducer/transducer_encoder/inputs_embedding/ convolution_stack/conv2d/kernel:0'
.
The error code tells you this element is referring to a non trackable element. It seems the non-trackable object is not directly assigned to an attribute of this conv2d/kernel:0.
To solve your issue, we need to localize Tensor("77040:0", shape=(), dtype=resource)
from this error code:
AssertionError: Tried to export a function which references untracked resource\
Tensor("77040:0", shape=(), dtype=resource).
TensorFlow objects (e.g. tf.Variable) captured by functions must be tracked by assigning them to an attribute of a tracked object or assigned to an attribute of the main object directly.
Thanks to your comments, we found that "ConvolutionStack" seems to reproduce the error.
The problem only occurs if I use the ConvolutionStack layer in InputsEmbedding but I can save both of them successfully in a standalone model.
I understand you cannot share the config of this layer and that's why I suggest you try and localize this Tensor(77040:0
from the ConvolutionStack.
This untrackable tensor must be an artifcat or a temporary tensor created by a process of a function of ConvolutionStack.
Try to find a tensor that could be passed from a function to another instead of being assigned to an attribute of a layer's class
QUESTION
Stopping and starting a deep learning google cloud VM instance causes tensorflow to stop recognizing GPU
Asked 2021-Jul-18 at 15:05I am using the pre-built deep learning VM instances offered by google cloud, with an Nvidia tesla K80 GPU attached. I choose to have Tensorflow 2.5 and CUDA 11.0 automatically installed. When I start the instance, everything works great - I can run:
Import tensorflow as tf
tf.config.list_physical_devices()
And my function returns the CPU, accelerated CPU, and the GPU. Similarly, if I run tf.test.is_gpu_available()
, the function returns True.
However, if I log out, stop the instance, and then restart the instance, running the same exact code only sees the CPU and tf.test.is_gpu_available()
results in False. I get an error that looks like the driver initialization is failing:
E tensorflow/stream_executor/cuda/cuda_driver.cc:355] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
Running nvidia-smi shows that the computer still sees the GPU, but my tensorflow can’t see it.
Does anyone know what could be causing this? I don’t want to have to reinstall everything when I’m restarting the instance.
ANSWER
Answered 2021-Jun-25 at 09:11Some people (sadly not me) are able to resolve this by setting the following at the beginning of their script/main:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
I had to reinstall CUDA drivers and from then on it worked even after restarting the instance. You can configure your system settings on NVIDIAs website and it will provide you the commands you need to follow to install cuda. It also asks you if you want to uninstall the previous cuda version (yes!).This is luckily also very fast.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
No vulnerabilities reported
Save this library and start creating your kit
HTTPS
https://github.com/tensorflow/tensorflow.git
CLI
gh repo clone tensorflow/tensorflow
SSH
git@github.com:tensorflow/tensorflow.git
Share this Page
See Similar Libraries in
by tensorflow
by ytdl-org
by tensorflow
by pytorch
by keras-team
See all Machine Learning Libraries
by tensorflow Python
by tensorflow TypeScript
by tensorflow Python
by tensorflow TypeScript
by tensorflow TypeScript
See all Libraries by this author
by ytdl-org
by scikit-learn
by tensorflow
by tensorflow
by keras-team
See all Machine Learning Libraries
by yangliuy
by asanoja
by allr
by mrmans0n
by LearnLib
See all Machine Learning Libraries
by firepick1
by vpejovic
by yangliuy
by asanoja
by allr
See all Machine Learning Libraries
by mrmans0n
by Credntia
by gatagat
by Hvass-Labs
by viadee
See all Machine Learning Libraries
Save this library and start creating your kit
Open Weaver – Develop Applications Faster with Open Source