kandi background
Explore Kits

vespa | The open big data serving engine. https://vespa.ai

 by   vespa-engine Java Version: v7.574.71 License: Apache-2.0

 by   vespa-engine Java Version: v7.574.71 License: Apache-2.0

Download this library from

kandi X-RAY | vespa Summary

vespa is a Java library typically used in Big Data, Tensorflow, Docker, Spark applications. vespa has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub, Maven.
The open big data serving engine - Store, search, organize and make machine-learned inferences over big data at serving time. This is the primary repository for Vespa where all development is happening. New production releases from this repository's master branch are made each weekday from Monday through Thursday.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • vespa has a highly active ecosystem.
  • It has 3879 star(s) with 473 fork(s). There are 151 watchers for this library.
  • There were 10 major release(s) in the last 6 months.
  • There are 119 open issues and 409 have been closed. On average issues are closed in 17 days. There are 29 open pull requests and 0 closed requests.
  • It has a positive sentiment in the developer community.
  • The latest version of vespa is v7.574.71
vespa Support
Best in #Java
Average in #Java
vespa Support
Best in #Java
Average in #Java

quality kandi Quality

  • vespa has no bugs reported.
vespa Quality
Best in #Java
Average in #Java
vespa Quality
Best in #Java
Average in #Java

securitySecurity

  • vespa has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
vespa Security
Best in #Java
Average in #Java
vespa Security
Best in #Java
Average in #Java

license License

  • vespa is licensed under the Apache-2.0 License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
vespa License
Best in #Java
Average in #Java
vespa License
Best in #Java
Average in #Java

buildReuse

  • vespa releases are available to install and integrate.
  • Deployable package is available in Maven.
  • Build file is available. You can build the component from source.
  • Installation instructions, examples and code snippets are available.
vespa Reuse
Best in #Java
Average in #Java
vespa Reuse
Best in #Java
Average in #Java
Top functions reviewed by kandi - BETA

kandi has reviewed vespa and discovered the below as its top functions. This is intended to give you an instant insight into vespa implemented functionality, and help decide if they suit your requirements.

  • Recursivelyurse through all tenants recursively .
  • Instantiates a URI item .
  • Convert a YARL expression tree into VESPA operators .
  • Export deployments from a collection of deployments .
  • Converts a GroupingExpression into an ExpressionNode
  • Write the definition of a class
  • Convert an indexed block to a StringBuilder
  • Returns the metrics for the search node .
  • Writes the logs in the specified time range to the specified output stream .
  • Reads a string from the input .

vespa Key Features

Home page: https://vespa.ai

Documentation: https://docs.vespa.ai

Continuous build: https://factory.vespa.oath.cloud

Run applications in the cloud for free: https://cloud.vespa.ai

Build Java modules

copy iconCopydownload iconDownload
export MAVEN_OPTS="-Xms128m -Xmx1024m"
./bootstrap.sh java
mvn install --threads 1C

Starting and stopping services in vespa

copy iconCopydownload iconDownload
curl --header Content-Type:application/zip --data-binary @target/application.zip   localhost:19071/application/v2/tenant/default/prepareandactivate |jq . 
{
    "log": [
        {
            "time": 1645036778830,
            "level": "WARNING",
            "message": "Change(s) between active and new application that require restart:\nIn cluster 'mycluster' of type 'search':\n    Restart services of type 'searchnode' because:\n        1) # Number of threads used per search\nproton.numthreadspersearch has changed from 2 to 5\n"
        }
    ],
    "tenant": "default",
    "url": "http://localhost:19071/application/v2/tenant/default/application/default/environment/prod/region/default/instance/default",
    "message": "Session 8 for tenant 'default' prepared and activated.",
    "configChangeActions": {
        "restart": [
            {
                "clusterName": "mycluster",
                "clusterType": "search",
                "serviceType": "searchnode",
                "messages": [
                    "# Number of threads used per search\nproton.numthreadspersearch has changed from 2 to 5"
                ],
                "services": [
                    {
                        "serviceName": "searchnode",
                        "serviceType": "searchnode",
                        "configId": "mycluster/search/cluster.mycluster/0",
                        "hostName": "vespa-container"
                    }
                ]
            }
        ],
        "refeed": [],
        "reindex": []
    }
}

Queries are very slow in local vespa

copy iconCopydownload iconDownload
 field address__postal_code type string {
            indexing: attribute
            attribute:fast-search
            weight : 70
}
-----------------------
schema doc {
  document doc {
   field license type string {
      indexing: summary | attribute
   }
   field title type string {
      indexing: summary | index
   }
   }
   rank-profile my-ranking {
     first-phase { expression: nativeRank(title) }
   }
} 
/search/?query=license:unk&yql=select id,license from doc where userQuery();&ranking=my-ranking
 field license type string {
      indexing: summary | attribute
      attribute:fast-search
 }
/search/?query=license:unk title:foo&yql=select id,license from doc where userQuery();&ranking=my-ranking
-----------------------
schema doc {
  document doc {
   field license type string {
      indexing: summary | attribute
   }
   field title type string {
      indexing: summary | index
   }
   }
   rank-profile my-ranking {
     first-phase { expression: nativeRank(title) }
   }
} 
/search/?query=license:unk&yql=select id,license from doc where userQuery();&ranking=my-ranking
 field license type string {
      indexing: summary | attribute
      attribute:fast-search
 }
/search/?query=license:unk title:foo&yql=select id,license from doc where userQuery();&ranking=my-ranking
-----------------------
schema doc {
  document doc {
   field license type string {
      indexing: summary | attribute
   }
   field title type string {
      indexing: summary | index
   }
   }
   rank-profile my-ranking {
     first-phase { expression: nativeRank(title) }
   }
} 
/search/?query=license:unk&yql=select id,license from doc where userQuery();&ranking=my-ranking
 field license type string {
      indexing: summary | attribute
      attribute:fast-search
 }
/search/?query=license:unk title:foo&yql=select id,license from doc where userQuery();&ranking=my-ranking
-----------------------
schema doc {
  document doc {
   field license type string {
      indexing: summary | attribute
   }
   field title type string {
      indexing: summary | index
   }
   }
   rank-profile my-ranking {
     first-phase { expression: nativeRank(title) }
   }
} 
/search/?query=license:unk&yql=select id,license from doc where userQuery();&ranking=my-ranking
 field license type string {
      indexing: summary | attribute
      attribute:fast-search
 }
/search/?query=license:unk title:foo&yql=select id,license from doc where userQuery();&ranking=my-ranking

`vespa` tutorial : ./src/python/user_search.py U33527 10 KeyError: 'children'

copy iconCopydownload iconDownload
$ java -jar vespa-http-client-jar-with-dependencies.jar \
       --file mind/vespa_user_embeddings.json \
       --endpoint http://localhost:8080
$ java -jar vespa-http-client-jar-with-dependencies.jar \
       --file mind/vespa_news_embeddings.json \
       --endpoint http://localhost:8080

Mismatch between content node storage ratio and server storage

copy iconCopydownload iconDownload
memory on node 0 [vespa-container] (0.802 > 0.800))

Command works inside a docker container but fails with docker exec

copy iconCopydownload iconDownload
docker exec vespa bash -c "/opt/vespa/bin/vespa-stop-services && /opt/vespa/bin/vespa-remove-index -force && /opt/vespa/bin/vespa-start-services"
docker exec vespa bash -c "/opt/vespa/bin/vespa-stop-services && /opt/vespa/bin/vespa-remove-index -force && /opt/vespa/bin/vespa-start-services"
Executing /opt/vespa/libexec/vespa/stop-vespa-base.sh
config-sentinel was running with pid 7788, sending SIGTERM
Waiting for exit (up to 15 minutes)
.. DONE
configproxy was running with pid 7666, sending SIGTERM
Waiting for exit (up to 15 minutes)
. DONE
[info] You have 23088 kilobytes of data for cluster msmarco
[info] For cluster msmarco distribution key 0 you have:
[info] 23084 kilobytes of data in var/db/vespa/search/cluster.msmarco/n0
[info] removing data:  rm -rf var/db/vespa/search/cluster.msmarco/n0
[info] removed.

Running /opt/vespa/libexec/vespa/start-vespa-base.sh
Starting config proxy using tcp/localhost:19070 as config source(s)
runserver(configproxy) running with pid: 10553
Waiting for config proxy to start
config proxy started after 1s (runserver pid 10553)
runserver(config-sentinel) running with pid: 10679
-----------------------
docker exec vespa bash -c "/opt/vespa/bin/vespa-stop-services && /opt/vespa/bin/vespa-remove-index -force && /opt/vespa/bin/vespa-start-services"
docker exec vespa bash -c "/opt/vespa/bin/vespa-stop-services && /opt/vespa/bin/vespa-remove-index -force && /opt/vespa/bin/vespa-start-services"
Executing /opt/vespa/libexec/vespa/stop-vespa-base.sh
config-sentinel was running with pid 7788, sending SIGTERM
Waiting for exit (up to 15 minutes)
.. DONE
configproxy was running with pid 7666, sending SIGTERM
Waiting for exit (up to 15 minutes)
. DONE
[info] You have 23088 kilobytes of data for cluster msmarco
[info] For cluster msmarco distribution key 0 you have:
[info] 23084 kilobytes of data in var/db/vespa/search/cluster.msmarco/n0
[info] removing data:  rm -rf var/db/vespa/search/cluster.msmarco/n0
[info] removed.

Running /opt/vespa/libexec/vespa/start-vespa-base.sh
Starting config proxy using tcp/localhost:19070 as config source(s)
runserver(configproxy) running with pid: 10553
Waiting for config proxy to start
config proxy started after 1s (runserver pid 10553)
runserver(config-sentinel) running with pid: 10679

Does Vespa support field projection for selected retrieval?

copy iconCopydownload iconDownload
document foo {
   field a ..
   field b ..
   field c ..
}

document-summary ab {
   field a .. { source: a}
   field b .. { source: b}
}

Is there a way of performing a weighted elementSimilarity with Vespa?

copy iconCopydownload iconDownload
field colors type weightedset<string>{
   indexing: summary | index
   match: text #This is default matching for string fields with 'index'
}
"colors": {
   "blue":1,
   "black and purple":1, 
   "green": 1,
   "yellow": 1
}
{
"yql": "select * from sources * where colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\");",
"ranking.profile": "color-ranking"
}
rank-profile color-ranking {
  function colorMatch() {
     expression: nativeDotProduct(colors)
  }
  first-phase {
   expression: colorMatch()
  } 
}
{
"yql": "select * from sources * where rank(foo contains "bar", colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\"));",
"ranking.profile": "color-ranking"
}
-----------------------
field colors type weightedset<string>{
   indexing: summary | index
   match: text #This is default matching for string fields with 'index'
}
"colors": {
   "blue":1,
   "black and purple":1, 
   "green": 1,
   "yellow": 1
}
{
"yql": "select * from sources * where colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\");",
"ranking.profile": "color-ranking"
}
rank-profile color-ranking {
  function colorMatch() {
     expression: nativeDotProduct(colors)
  }
  first-phase {
   expression: colorMatch()
  } 
}
{
"yql": "select * from sources * where rank(foo contains "bar", colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\"));",
"ranking.profile": "color-ranking"
}
-----------------------
field colors type weightedset<string>{
   indexing: summary | index
   match: text #This is default matching for string fields with 'index'
}
"colors": {
   "blue":1,
   "black and purple":1, 
   "green": 1,
   "yellow": 1
}
{
"yql": "select * from sources * where colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\");",
"ranking.profile": "color-ranking"
}
rank-profile color-ranking {
  function colorMatch() {
     expression: nativeDotProduct(colors)
  }
  first-phase {
   expression: colorMatch()
  } 
}
{
"yql": "select * from sources * where rank(foo contains "bar", colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\"));",
"ranking.profile": "color-ranking"
}
-----------------------
field colors type weightedset<string>{
   indexing: summary | index
   match: text #This is default matching for string fields with 'index'
}
"colors": {
   "blue":1,
   "black and purple":1, 
   "green": 1,
   "yellow": 1
}
{
"yql": "select * from sources * where colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\");",
"ranking.profile": "color-ranking"
}
rank-profile color-ranking {
  function colorMatch() {
     expression: nativeDotProduct(colors)
  }
  first-phase {
   expression: colorMatch()
  } 
}
{
"yql": "select * from sources * where rank(foo contains "bar", colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\"));",
"ranking.profile": "color-ranking"
}
-----------------------
field colors type weightedset<string>{
   indexing: summary | index
   match: text #This is default matching for string fields with 'index'
}
"colors": {
   "blue":1,
   "black and purple":1, 
   "green": 1,
   "yellow": 1
}
{
"yql": "select * from sources * where colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\");",
"ranking.profile": "color-ranking"
}
rank-profile color-ranking {
  function colorMatch() {
     expression: nativeDotProduct(colors)
  }
  first-phase {
   expression: colorMatch()
  } 
}
{
"yql": "select * from sources * where rank(foo contains "bar", colors contains ([{\"weight\":1}]\"purple\") or colors contains ([{\"weight\":2}]\"yellow\"));",
"ranking.profile": "color-ranking"
}

How to include a OneHot in an ONNX coming from PyTorch

copy iconCopydownload iconDownload
function oneHotInput() {
    expression: tensor(x[10])(x == attribute(myInteger))
}
-----------------------
#! /usr/bin/env python3

import torch
import torch.onnx
import torch.nn.functional as F


class MyModel(torch.nn.Module):
    def __init__(self, classes=5):
        super(MyModel, self).__init__()
        self._classes = classes
        self.linear = torch.nn.Linear(in_features=self._classes, out_features=1)
        self.logistic = torch.nn.Sigmoid()

    def forward(self, input):
        one_hot = F.one_hot(input, num_classes=self._classes).float()
        return self.logistic(self.linear(one_hot))


def main():
    model = MyModel()

    # training omitted

    data = torch.tensor([0, 4, 2])
    torch.onnx.export(model, data, "test.onnx", 
        input_names=["input"], output_names=["output"])
        
    result = model.forward(data)  
    print(result)

if __name__ == "__main__":
    main()
tensor([[0.5749],
        [0.5081],
        [0.5581]], grad_fn=<SigmoidBackward>)
In [1]: import onnxruntime as ort                                                                                                                                                            
In [2]: m = ort.InferenceSession("test.onnx")                                                                                                                                                
In [3]: m.run(input_feed={"input":[0,4,2]}, output_names=["output"])                                                                                                                        
Out[3]: 
[array([[0.57486993],
        [0.5081395 ],
        [0.5580716 ]], dtype=float32)]
-----------------------
#! /usr/bin/env python3

import torch
import torch.onnx
import torch.nn.functional as F


class MyModel(torch.nn.Module):
    def __init__(self, classes=5):
        super(MyModel, self).__init__()
        self._classes = classes
        self.linear = torch.nn.Linear(in_features=self._classes, out_features=1)
        self.logistic = torch.nn.Sigmoid()

    def forward(self, input):
        one_hot = F.one_hot(input, num_classes=self._classes).float()
        return self.logistic(self.linear(one_hot))


def main():
    model = MyModel()

    # training omitted

    data = torch.tensor([0, 4, 2])
    torch.onnx.export(model, data, "test.onnx", 
        input_names=["input"], output_names=["output"])
        
    result = model.forward(data)  
    print(result)

if __name__ == "__main__":
    main()
tensor([[0.5749],
        [0.5081],
        [0.5581]], grad_fn=<SigmoidBackward>)
In [1]: import onnxruntime as ort                                                                                                                                                            
In [2]: m = ort.InferenceSession("test.onnx")                                                                                                                                                
In [3]: m.run(input_feed={"input":[0,4,2]}, output_names=["output"])                                                                                                                        
Out[3]: 
[array([[0.57486993],
        [0.5081395 ],
        [0.5580716 ]], dtype=float32)]
-----------------------
#! /usr/bin/env python3

import torch
import torch.onnx
import torch.nn.functional as F


class MyModel(torch.nn.Module):
    def __init__(self, classes=5):
        super(MyModel, self).__init__()
        self._classes = classes
        self.linear = torch.nn.Linear(in_features=self._classes, out_features=1)
        self.logistic = torch.nn.Sigmoid()

    def forward(self, input):
        one_hot = F.one_hot(input, num_classes=self._classes).float()
        return self.logistic(self.linear(one_hot))


def main():
    model = MyModel()

    # training omitted

    data = torch.tensor([0, 4, 2])
    torch.onnx.export(model, data, "test.onnx", 
        input_names=["input"], output_names=["output"])
        
    result = model.forward(data)  
    print(result)

if __name__ == "__main__":
    main()
tensor([[0.5749],
        [0.5081],
        [0.5581]], grad_fn=<SigmoidBackward>)
In [1]: import onnxruntime as ort                                                                                                                                                            
In [2]: m = ort.InferenceSession("test.onnx")                                                                                                                                                
In [3]: m.run(input_feed={"input":[0,4,2]}, output_names=["output"])                                                                                                                        
Out[3]: 
[array([[0.57486993],
        [0.5081395 ],
        [0.5580716 ]], dtype=float32)]

Moving cell data to respective row based on numbers in another column

copy iconCopydownload iconDownload
Const filepath As String = "C:\path\to\excel\file.xlsx"
Const sheetname As String = "Sheet1"

Dim connectionString As String
connectionString = _
    "Provider=Microsoft.ACE.OLEDB.12.0;" & _
    "Data Source=""" & filepath & """;" & _
    "Extended Properties=""Excel 12.0;HDR=No"""

    ' If your data is in a macro-enabled file (.xlsm), the previous line should
    ' look like this:
    ' "Extended Properties=""Excel 12.0 Macro;HDR=No"""

Dim sql As String
sql = _
    "SELECT F1, F2, F3, F4, F5 " & _
    "FROM (" & _
        "SELECT F1, CInt(Mid(F1, 2)) AS F1_1 " & _
        "FROM [Sheet1$]" & _
    ") AS t1 " & _
    "LEFT JOIN (" & _
        "SELECT F2, F3, F4, F5 FROM [Sheet1$]" & _
    ") AS t2 ON t1.F1_1 = t2.F2"

Dim rs As New ADODB.Recordset
rs.Open sql, connectionString

Worksheets("Sheet2").Range("A1").CopyFromRecordset(rs)

Vespa.ai: How to make an array of floats handle null values?

copy iconCopydownload iconDownload
field stuff type tensor<float>(x{}) {
  indexing: summary |attribute
}
[
  { "put": "id:example:example::0", "fields": {
      "stuff" : {
        "cells": [
          { "address" : { "x" : "0" }, "value": 2.0 },
          { "address" : { "x" : "1" }, "value": 1.4 },
          { "address" : { "x" : "3" }, "value": 5.6 },
        
        ]
      }
    }
  }
]
-----------------------
field stuff type tensor<float>(x{}) {
  indexing: summary |attribute
}
[
  { "put": "id:example:example::0", "fields": {
      "stuff" : {
        "cells": [
          { "address" : { "x" : "0" }, "value": 2.0 },
          { "address" : { "x" : "1" }, "value": 1.4 },
          { "address" : { "x" : "3" }, "value": 5.6 },
        
        ]
      }
    }
  }
]

Community Discussions

Trending Discussions on vespa
  • Dynamic addition of fields in vespa
  • Starting and stopping services in vespa
  • How to downgrade the version of the vespa engine installed in ec2 server
  • Versioning document changes in Vespa
  • Queries are very slow in local vespa
  • `vespa` tutorial : ./src/python/user_search.py U33527 10 KeyError: 'children'
  • Is there a way to reuse a struct in Vespa
  • Vespa response: Summary data is incomplete: Timed out waiting for summary data
  • Mismatch between content node storage ratio and server storage
  • Vespa.ai exploit multiple instances to answer queries
Trending Discussions on vespa

QUESTION

Dynamic addition of fields in vespa

Asked 2022-Mar-14 at 09:55

In Elastic Search, to add new fields while running the application we have to provide

"dynamic":true

More info about the same: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html

Is there any functionality which can replicate same behaviour in Vespa? I was not able to find in vespa documentation. Kindly help me in this regard. Thank you.

ANSWER

Answered 2022-Mar-14 at 09:47

https://docs.vespa.ai/en/schemas.html#schema-modifications is the best place to start - just modify the schema with new fields and redeploy the application. The new fields can not have a default value, they are empty. It is not necessary to restart Vespa, this can be done on a running instance.

Source https://stackoverflow.com/questions/71465539

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install vespa

Run your own Vespa instance: https://docs.vespa.ai/en/getting-started.html Or deploy your Vespa applications to the cloud service: https://cloud.vespa.ai.
Use this if you only need to build the Java modules, otherwise follow the complete development guide above.
See Building Vespa RPM for details.

Support

We welcome contributions! See CONTRIBUTING.md to learn how to contribute. If you want to contribute to the documentation, see https://github.com/vespa-engine/documentation.

DOWNLOAD this Library from

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Share this Page

share link
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

  • © 2022 Open Weaver Inc.