capistrano | Remote multi-server automation tool | Continuous Deployment library
kandi X-RAY | capistrano Summary
Support
Quality
Security
License
Reuse
- Adds a filter to the server .
- Invokes a task .
- Define a file
- Execute a shell call .
- Configures TLS configuration .
- Returns true if the value exists .
- Setup the command .
- Instantiates a new instance .
- Loads the dependencies of the load
- Defines a task .
capistrano Key Features
capistrano Examples and Code Snippets
Trending Discussions on capistrano
Trending Discussions on capistrano
QUESTION
So I have a file not found problem. I have an engine that works in development mode in the engines test/dummy app, the engine allows the editing of sass variables and stores them in a theme table, the variables are used by a sass partial such as _banner.scss containing variables used in the main stylesheet such as $banner_color which is then imported into the main stylesheet which in turn is precompiled using an initializer in the engine.rb file and inclusion in the app/config/engine_name_manifest.js.
The files are all available in development with the local dummy app but not in the eventual host app due to the assets being compiled.
I have a rake task that takes the data, updates the relevant partial e.g. _banner.scss with the data from the theme table but of course the partials are not not available in a host app as the engine has already compiled them. I'm looking for a solution that will allow me to edit the raw, uncompiled stylesheets then recompile them. Obviously my Capistrano deploy script will need to reapply the stylesheet changes every deployment but that is just a rake task call. What approach should I take? Should I find a way to copy the css files to the host app in an engine initializer? Should I use a different approach entirely, I have started looking at propshaft but that is a massive step to replace sass rails and I'm not sure how that would help
The engine
require "deface"
require 'ccs_cms_admin_dashboard'
require 'ccs_cms_custom_page'
require 'ccs_cms_core'
require 'css_menu'
#require 'tinymce-rails'
require 'delayed_job_active_record'
require 'daemons'
require 'sprockets/railtie'
require 'sassc-rails'
module CcsCms
module PublicTheme
class Engine < ::Rails::Engine
isolate_namespace CcsCms::PublicTheme
paths["app/views"] << "app/views/ccs_cms/public_theme"
initializer "ccs_cms.assets.precompile" do |app|
app.config.assets.precompile += %w( public_theme_manifest.js )
end
initializer :append_migrations do |app|
unless app.root.to_s.match?(root.to_s)
config.paths['db/migrate'].expanded.each do |p|
app.config.paths['db/migrate'] << p
end
end
end
initializer :active_job_setup do |app|
app.config.active_job.queue_adapter = :delayed_job
end
config.to_prepare do
Dir.glob(Engine.root.join("app", "decorators", "**", "*_decorator*.rb")) do |c|
Rails.configuration.cache_classes ? require(c) : load(c)
end
end
config.generators do |g|
g.test_framework :rspec,
fixtures: false,
request: false,
view_specs: false,
helper_specs: false,
controller_specs: false,
routing_specs: false
g.fixture_replacement :factory_bot
g.factory_bot dir: 'spec/factories'
end
end
end
end
The Css class that writes the css
class Css
def get_stylesheet_path
Rails.root.join("app/assets/stylesheets/ccs_cms/public_theme")
end
def write_css(theme)
update_css_files_for(theme.banner, '_public_banner.scss', BANNER_ARRAY, BANNER_FIELD_MAP)
update_css_files_for(theme.banner.font, '_public_banner_font.scss', BANNER_FONT_ARRAY, BANNER_FONT_FIELD_MAP)
end
private
def update_css_files_for(model_record_to_use, css_file, array_to_use, field_map)
amended_css = amend_css_for(model_record_to_use, css_file, array_to_use, field_map)
create_css_files_for(css_file, amended_css)
end
def amend_css_for(model_record_to_use, file_name, array_to_use, field_map)
original_css_array = IO.readlines("#{get_stylesheet_path}/#{file_name}")
new_array = []
original_css_array.each do |line|
new_line = line
array_to_use.each do |ma|
if line.start_with?(ma)
field_name = field_map[ma.to_sym]
new_line = ma + ": #{model_record_to_use[field_name.to_sym]};"
#puts("@@@@ original line: #{line}, ma: #{ma}, Field name: #{field_name}, value: #{theme[field_name]}")
break
end
end
new_array << new_line
end
new_array
end
# ---- File and I/O Handling ---- #
def create_css_files_for(file_name, css_array)
File.open("#{get_stylesheet_path}/#{file_name}", "w") do |file|
file.puts css_array
end
end
end
ANSWER
Answered 2022-Apr-02 at 03:44Thanks for clarifying. If I understood correctly here my take on it.
partials are not not available in a host app as the engine has already compiled them
Partials are still there, precompilation just outputs *.{css/js}
files into public/assets/
that are declared in app/assets/config/manifest.js
.
To get to engines files, instead of Rails.root
use:
CcsCms::PublicTheme::Engine.root
In Css
class, for example:
def get_stylesheet_path
CcsCms::PublicTheme::Engine.root.join("app/assets/stylesheets/ccs_cms/public_theme")
end
To support changing theme engines. Theme root can be set in an engine initializer to something like Rails.configuration.theme_root
and used in the main app.
Because your theme is also configurable, I think it's better to read theme's original sass files but not modify them, copy them into a tmp folder and update with values from theme table, then output a theme.css
in the main app with sass. https://sass-lang.com/documentation/cli/dart-sass
# Compiles all Sass
$ sass tmp/theme/application.scss:app/stylesheets/theme.css
Then let Rails take over the precompilation process.
Another option is to have one sass configuration file and only update this file. That way there is no dependency on the file structure of any particular theme.
import 'configuration' // sass variables with values from theme table
import 'banner' // uses sass variables only
...
Also just use css variables, if that's an option, and avoid all of the above complexity; no precompilation, no redeploys when theme table changes.
Update for css variables.
Just so we're on the same page. I meant these css variables: https://developer.mozilla.org/en-US/docs/Web/CSS/Using_CSS_custom_properties. If Internet Explorer is not a priority for you, this is the best solution. The setup is something like this:
<%= stylesheet_link_tag 'application' %>
/* app/assets/stylesheets/application.css */
p { color: var(--text-color); }
Possible fix to avoid amending css files. Use erb
interpolation inside the sass files. No need to amend every time a theme configuration is changed. In development compiles on the fly. In production it has to be precompiled again when theme configuration is changed; no amending.
// _banner.scss.erb
p { color: <%= Theme.text_color %>; }
You could even use amend_css_for
function to insert literal erb code and save some time. For example
new_line = ma + ": <%= Theme.#{model_name}.#{field_name} %>;"
Finally, if you don't want to touch engine files and because these files are not part of the main/host app (as in literally two separate folders in the filesystem). You have to make a copy when amending; read from CcsCms::PublicTheme::Engine.root
write to Rails.root
.
def get_stylesheet_path
CcsCms::PublicTheme::Engine.root.join("app/assets/stylesheets/ccs_cms/public_theme")
end
# but save to main app
def create_css_files_for(file_name, css_array)
File.open("#{Rails.root.join("app/assets/stylesheets/ccs_cms/public_theme")}/#{file_name}", "w") do |file|
file.puts css_array
end
end
QUESTION
I have nginx + nginx unit + django python application , and django project is is deployed by capistrano
deploy.rb
lock "~> 3.16.0"
set :application, "mynavi"
set :branch, 'master'
set :deploy_to, "/var/www/html/mynavi"
set :linked_dirs, fetch(:linked_dirs, []).push('static')
set :keep_releases, 3
set :linked_files, %w{.env}
set :repo_url, "ubuntu@git.example.jp:/~/myGit/mynavi.git"
production.rb
set :stage, :production
set :branch, 'master'
server 'lion.example.jp', user: 'ubuntu', roles: %w(app), primary: true
namespace :deploy do
desc 'Collec Static Files'
task :collectImg do
on roles(:app) do
execute "source activate mynavi;/home/ubuntu/anaconda3/envs/mynavi/bin/python /var/www/html/mynavi/current/manage.py collectstatic --noinput"
end
end
after :publishing, :collectImg
end
cap prodution deploy
makes deployment successfully
However, after deployment
I need to restart unit
manually.
sudo systemctl restart unit
Can I do this automatic after deployment?
Solution
Thanks to @Timo Stark 's answer
My final production.rb is here, just adding the curl line.
production.rb
set :stage, :production
set :branch, 'master'
server 'lion.example.jp', user: 'ubuntu', roles: %w(app), primary: true
namespace :deploy do
desc 'Collec Static Files'
task :collectImg do
on roles(:app) do
execute "source activate mynavi;/home/ubuntu/anaconda3/envs/mynavi/bin/python /var/www/html/mynavi/current/manage.py collectstatic --noinput"
execute "sudo curl -X GET --unix-socket /path/to/control.unit.sock http://localhost/control/applications/app_name/restart"
end
end
after :publishing, :collectImg
end
ANSWER
Answered 2022-Feb-25 at 17:46With the latest release you are able to restart the application and let Unit reload the code from the workdir.
sudo curl -X GET --unix-socket /path/to/control.unit.sock \
http://localhost/control/applications/app_name/restart
https://unit.nginx.org/configuration/#process-management
Happy to discuss the capistrano intergration a little further.
QUESTION
I upgrade my ruby version to 2.6.5. I deployed it to my server using capistrano.
But my nginx logs say this:
App 9470 output: /bin/sh: 1: exec: /home/deploy/.rvm/gems/ruby-2.3.1/wrappers/ruby: not found
[ E 2022-01-27 12:34:23.7336 9450/Tc age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /home/deploy/taddar/current: The application process exited prematurely.
Error ID: d1f83ca0
Error details saved to: /tmp/passenger-error-0KwZUf.html
[ E 2022-01-27 12:34:23.7393 9450/T9 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is d1f83ca0. Please see earlier logs for details about the error.
When I run ruby -v
I get 2.6.5, yet above you can see its looking for 2.3.1:
ruby -v
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
Any ideas on how to fix this?
In my deploy.rb I set the ruby version.
set :rvm_ruby_version, '2.6.5'
My nginx.config looks like:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
passenger_ruby /usr/bin/ruby2.6.5;
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
underscores_in_headers on;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/vnd.ms-fontobject application/x-font-ttf font/opentype image/svg+xml image/x-icon;
##
# Phusion Passenger config
##
# Uncomment it if you installed passenger or passenger-enterprise
##
include /etc/nginx/passenger.conf;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
ANSWER
Answered 2022-Jan-27 at 22:58Bingo got it working. Thanks to @razvans and @engineersmnky for pointing me in the right direction.
Yes I had references to passenger_ruby
but it was in the wrong place. I had to go to /etc/nginx/sites-available
and add passenger_ruby /path/to/ruby
To find out what the /path/to/ruby
is use passenger-config about ruby-command
and use the value at Command
.
passenger-config about ruby-command
passenger-config was invoked through the following Ruby interpreter:
Command: /home/deploy/.rvm/gems/ruby-2.6.5/wrappers/ruby
So mine was
server {
....
passenger_ruby /home/deploy/.rvm/gems/ruby-2.6.5/wrappers/ruby
You might want to know any other references you have to ruby so they don't conflict with each other. A useful command is: grep -rnw 'path' -e 'passenger_ruby'
This doc helped me a lot https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/#determine_ruby_command
QUESTION
On a brand new digitalocean droplet running Ubuntu 20.10 with a brand new pretty near empty rails 7 alpha 2 app running bundle install
results in the following both when running cap production deploy on my local machine and when running from the command shell on the droplet
Fetching gem metadata from https://rubygems.org/............
Could not find turbo-rails-7.1.1 in any of the sources
I have no issues locally with this so this has to be an environment issue on the production server or some configuration option I've missed somehere. I am using latest stable rvm which is correctly configured,
rvm gemset list
gemsets for ruby-3.0.0 (found in /home/comtechmaster/.rvm/gems/ruby-3.0.0)
(default)
global
=> master_cms
I used a capistrano script to deploy the app and latest stable version node was installed using nvm. Bundler is the same version on both environments and turbo-rails-7.1.1 does not appear anywhere in my gemfile.lock
ANSWER
Answered 2021-Nov-09 at 14:37I ran into this also. Not sure why, but they yanked the 7.x versions and regressed to 0.8.x:
https://rubygems.org/gems/turbo-rails/versions/7.1.1
Just add this to your Gemfile:
gem 'turbo-rails', '~> 0.8'
And then run and run ./bin/bundle update turbo-rails
and you should be good.
QUESTION
Currently, Rails application is on service.
The environment is following the below.
I'm thinking to introduce React to my app.
But in case of management screen, I want to use Rails.
Because it's already customized a lot and no complaint so far.
I think to introduce React to Rails app is 3way.
- Using Webpacker
- Using Gem(react-rails/react_on_rails)
- Using
create-react-app
From my thinking, 1&2 and 3 have big differences. 1&2 is React renders from Rails View. 3 is React renders independently and Rails is just used for API.
So I have three questions.
- Compare 1&2 and 3, which is faster? Is there big different?
- In case of 3, I think React'host should be 3000, Rails should be 3001. Is it possible to release step by step? Like introducing React only top screen, others are Rails.
- In case of 3, can I use RailsAdmin for management screen?
I have no idea.
Please help me..
ANSWER
Answered 2022-Jan-14 at 08:33I would almost always use the approach number 3, as in using Rails as an API and having admin panel. To achieve this you simply define a different route for all the api endpoints and the admin, don't overthink this. Under the rails port you can put all the API endpoints and a rails admin endpoint. Using Webpacker in rails is still supported but starting with rails 7 it is no longer the default, which means that in case of some problems (which in my experience are frequent with webpacker) you might have a harder time finding some help. Using gems to work with JS frameworks always seems to me like an unnecessary complication, and adding an extra tool to help with a different tool...
Keep both tools to themselves, you can still have both apps in one repository nothing is stoping you from that, but Rails being responsible for the backend and react being the frontend is the most sensible approach IMHO. Nothing stands in a way of using rails app as both the API server and rails admin managment view. It is just a matter of setting up the routes.
QUESTION
My environment is following the below.
Environment Version Rails 7.0.0 Ruby 3.0.0 capistrano 3.16.0 Production environment Amazon EC2 LinuxAfter for while from deploying, I got these logs.
Then deploying stopped, not working anymore.
INFO [41bfeeb4] Running /usr/bin/env sudo /bin/systemctl stop puma_〇〇_production as deploy@〇〇
DEBUG [41bfeeb4] Command: ( export RBENV_ROOT="/usr/local/src/rbenv" RBENV_VERSION="3.0.0" ; /usr/bin/env sudo /bin/systemctl stop puma_〇〇_production )
DEBUG [41bfeeb4]
あなたはシステム管理者から通常の講習を受けたはずです。
これは通常、以下の3点に要約されます:
#1) 他人のプライバシーを尊重すること。
#2) タイプする前に考えること。
#3) 大いなる力には大いなる責任が伴うこと。
DEBUG [41bfeeb4] [sudo] deploy password:
I think it's related to authorization.
I have no idea for fixing.
How can I do?
deploy.rb
set :application, '〇〇'
set :repo_url, '〇〇'
set :deploy_to, '/var/www/〇〇'
set :puma_threads, [4, 16]
set :puma_workers, 0
set :pty, true
set :use_sudo, false
set :stage, :staging
set :deploy_via, :remote_cache
set :deploy_to, "/var/www/#{fetch(:application)}"
set :puma_bind,
"unix://#{shared_path}/tmp/sockets/#{fetch(:application)}-puma.sock"
set :puma_state, "#{shared_path}/tmp/pids/puma.state"
set :puma_pid, "#{shared_path}/tmp/pids/puma.pid"
set :puma_access_log, "#{release_path}/log/puma.access.log"
set :puma_error_log, "#{release_path}/log/puma.error.log"
set :puma_preload_app, true
set :puma_worker_timeout, nil
set :puma_init_active_record, true
set :puma_restart_command, 'bundle exec puma'
set :rbenv_type, :system
set :rbenv_path, '/usr/local/src/rbenv'
set :rbenv_ruby, '3.0.0'
set :linked_dirs,
fetch(:linked_dirs, []).push(
'log',
'tmp/pids',
'tmp/cache',
'tmp/sockets',
'vendor/bundle',
'public/system',
'public/uploads',
)
set :linked_files,
fetch(:linked_files, []).push(
'config/database.yml',
'config/secrets.yml',
'config/puma.rb',
'.env',
)
namespace :puma do
Rake::Task[:restart].clear_actions
desc 'Overwritten puma:restart task'
task :restart do
puts 'Overwriting puma:restart to ensure that puma is running. Effectively, we are just starting Puma.'
puts 'A solution to this should be found.'
invoke 'puma:stop'
invoke 'puma:start'
end
desc 'Create Directories for Puma Pids and Socket'
task :make_dirs do
on roles(:app) do
execute "mkdir #{shared_path}/tmp/sockets -p"
execute "mkdir #{shared_path}/tmp/pids -p"
end
end
before :start, :make_dirs
end
namespace :deploy do
desc 'Make sure local git is in sync with remote.'
task :check_revision do
on roles(:app) do
unless `git rev-parse HEAD` == `git rev-parse origin/master`
puts 'WARNING: HEAD is not the same as origin/master'
puts 'Run `git push` to sync changes.'
exit
end
end
end
desc 'Restart application'
task :restart do
on roles(:app), in: :sequence, wait: 5 do
invoke 'puma:restart'
end
end
before :starting, :check_revision
after :finishing, :compile_assets
after :finishing, :cleanup
end
after 'deploy', 'sitemap:refresh'
Capfile
require 'capistrano/setup'
require 'capistrano/deploy'
require 'capistrano/scm/git'
install_plugin Capistrano::SCM::Git
require 'capistrano/rails'
require 'capistrano/rbenv'
require 'capistrano/rails/assets'
require 'capistrano/rails/migrations'
require 'capistrano/bundler'
require 'capistrano/puma'
require 'capistrano/sitemap_generator'
require 'whenever/capistrano'
require 'dotenv'
Dotenv.load
install_plugin Capistrano::Puma
install_plugin Capistrano::Puma::Systemd
# Loads custom tasks from `lib/capistrano/tasks' if you have any defined.
Dir.glob('lib/capistrano/tasks/*.cap').each { |r| import r }
ANSWER
Answered 2022-Jan-07 at 17:12The deploy user needs to be a sudo user in order to restart the puma systemctl service.
You can fix the issue by creating new file inside /etc/sudoers.d directory and add this line
deploy ALL=(ALL) NOPASSWD:/bin/systemctl
QUESTION
I am working on a Ruby on Rails application that uses the gem Private Pub (https://github.com/ryanb/private_pub) to have a real time chat messages feature
This application is hosted in Digital Ocean and chat messages have been working correctly for a long time. However, around 1 and a half month ago it stopped working and I got an error on the console that says: GET https://fake-url.com:9292/faye/faye.js net::ERR_CONNECTION_REFUSED
First thing I did was to restart the Faye server using this command: bundle exec thin -C /data/app/shared/config/private_pub_thin.yml -d -P /data/app/shared/tmp/pids/faye.pid --ssl-disable-verify start (I run this command through Capistrano)
After running that command I checked /data/app/shared/tmp/pids folder and confirmed that faye.pid is there so process seems to be running
Then I ran sudo lsof -i -P -n | grep LISTEN to check open ports and 9292 port is listed
After that I reloaded the chat and got again the net::ERR_CONNECTION_REFUSED error
Then I checked the ports again and port 9292 does not appear on the list
So it seems like everytime I load the pages it closes the port or something like that.
NOTES:
- ufw is not enabled
- We have a staging server where we have the same problem. Only difference is that in staging port 9292 always appears on the list of open ports. However, it throws net::ERR_CONNECTION_REFUSED
- I think this error started to happen after the old Let’s Encrypt Root Certificate expired (https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021) I am not 100% sure about this but I had other problems related to this and I wonder if this error is somehow related to that
- It works fine locally
Thanks in advance!
ANSWER
Answered 2021-Dec-29 at 14:21I ended up removing Private Pub gem and using Pusher to get the real time updates. Pusher was really easy to use. Thank you all!
QUESTION
I'm trying to deploy ruby on rails application to the Amazon EC2 instance, using puma as an application server, nginx server and capistrano for deployments. Currently I'm stuck with the error message:
nginx logs are not showing any error,
puma logs are also not showing what's the error
I have verified that puma server is started and running, using the following command:
ubuntu@ip-nnnnn:~$ ps aux | grep puma
deployer 89555 0.0 1.5 652040 248180 ? Sl 21:48 0:00 puma 3.12.1 (unix://nnnn/nnn/current/tmp/sockets/puma.sock) [20211130214658]
deployer 89561 0.0 1.5 854804 259336 ? Sl 21:48 0:00 puma: cluster worker 0: 89555 [20211130214658]
deployer 89566 0.0 1.5 787216 247824 ? Sl 21:48 0:00 puma: cluster worker 1: 89555 [20211130214658]
ubuntu 89845 0.0 0.0 8168 740 pts/0 S+ 22:23 0:00 grep --color=auto puma
capistrano deploys without any error, all tasks are completed fine, the puma server is restarted fine during the capistrano deployment.
application log has no recent data, it looks like it doesn't reach to that point, it breaks before that.
ANSWER
Answered 2021-Dec-09 at 01:08In my particular case it was due to the stopped REDIS service. The application was configured to utilize REDIS for caching, however REDIS service wasn't running. This wasn't reflected in the logs for some reason and I spent a good amount of time figuring out what was wrong.
QUESTION
I have downloaded a list of all the towns and cities etc in the US from the census bureau. Here is a random sample:
dput(somewhere)
structure(list(state = structure(c(30L, 31L, 5L, 31L, 24L, 36L,
13L, 21L, 6L, 10L, 31L, 28L, 10L, 5L, 5L, 8L, 23L, 11L, 34L,
19L, 29L, 4L, 24L, 13L, 21L, 31L, 2L, 3L, 29L, 24L, 1L, 13L,
15L, 10L, 11L, 33L, 35L, 8L, 11L, 12L, 36L, 28L, 9L, 31L, 8L,
14L, 11L, 12L, 36L, 13L, 8L, 5L, 29L, 8L, 7L, 23L, 25L, 39L,
16L, 28L, 10L, 29L, 26L, 8L, 32L, 40L, 28L, 23L, 37L, 31L, 18L,
5L, 1L, 31L, 18L, 13L, 11L, 10L, 25L, 18L, 21L, 18L, 11L, 35L,
31L, 36L, 20L, 19L, 38L, 2L, 40L, 13L, 36L, 11L, 29L, 27L, 22L,
17L, 12L, 20L), .Label = c("ak", "al", "ar", "az", "ca", "co",
"fl", "ga", "hi", "ia", "il", "in", "ks", "ky", "la", "md", "mi",
"mo", "ms", "mt", "nc", "nd", "ne", "nj", "nm", "nv", "ny", "oh",
"ok", "or", "pa", "pr", "ri", "sc", "sd", "tx", "ut", "va", "wa",
"wi"), class = "factor"), geoid = c(4120100L, 4280962L, 668028L,
4243944L, 3460600L, 4871948L, 2046000L, 3747695L, 839965L, 1909910L,
4244824L, 3902204L, 1963750L, 622360L, 669088L, 1382972L, 3125230L,
1722736L, 4539265L, 2804705L, 4039625L, 451465L, 3467020L, 2077150L,
3765260L, 4221792L, 133904L, 566320L, 4033150L, 3463180L, 223460L,
2013675L, 2232405L, 1951600L, 1752142L, 4445010L, 4655684L, 1336416L,
1729080L, 1840842L, 4804672L, 3932207L, 1523675L, 4260384L, 1321912L,
2159232L, 1735307L, 1867176L, 4839304L, 2057350L, 1309656L, 655380L,
4082250L, 1350680L, 1275475L, 3147745L, 3505010L, 5352285L, 2483337L,
3909834L, 1912945L, 4068200L, 3227900L, 1366304L, 7286831L, 5505350L,
3982390L, 3149915L, 4974480L, 4249440L, 2943346L, 677430L, 280770L,
4247872L, 2902242L, 2039075L, 1735281L, 1932565L, 3580120L, 2973852L,
3722620L, 2943238L, 1755938L, 4643100L, 4251904L, 4830920L, 3056575L,
2801940L, 5156048L, 137000L, 5508925L, 2057300L, 4861172L, 1736477L,
4021200L, 3677783L, 3832060L, 2614900L, 1820332L, 3043750L),
ansicode = c(2410344L, 2390453L, 2411793L, 1214759L, 885360L,
2412035L, 485621L, 2403359L, 2412812L, 2583481L, 2390095L,
2397971L, 2396237L, 2585422L, 2411819L, 2405746L, 2398338L,
2394628L, 2812929L, 2586582L, 2408478L, 2582836L, 885393L,
2397270L, 2402898L, 2584453L, 2405811L, 2405518L, 2412737L,
2389752L, 2418574L, 2393549L, 2402559L, 2629970L, 2399453L,
2378109L, 2812999L, 2402563L, 2398956L, 2396699L, 2409759L,
2393028L, 2414061L, 2805542L, 2404192L, 2404475L, 2398514L,
2629884L, 2408486L, 2396265L, 2405306L, 2411363L, 2413515L,
2405064L, 2402989L, 2583899L, 2629103L, 2585016L, 2390487L,
2397481L, 2393811L, 2413298L, 2583927L, 2812702L, 2415078L,
1582764L, 2400116L, 2400036L, 2412013L, 2633665L, 2787908L,
2583158L, 2418866L, 1214943L, 2393998L, 485611L, 2398513L,
2394969L, 2806756L, 2397053L, 2406485L, 2395719L, 2399572L,
1267480L, 2389516L, 2410660L, 2409026L, 2806379L, 2584894L,
2404746L, 2586459L, 2396263L, 2411528L, 2398556L, 2412443L,
2584298L, 1036064L, 2806333L, 2396920L, 2804282L), city = c("donald",
"warminster heights", "san juan capistrano", "littlestown",
"port republic", "taylor", "merriam", "northlakes", "julesburg",
"california junction", "lower allen", "antwerp", "pleasantville",
"el rancho", "santa clarita", "willacoochee", "kennard",
"effingham", "la france", "beechwood", "keys", "orange grove mobile manor",
"shiloh", "west mineral", "stony point", "east salem", "heath",
"stamps", "haworth", "rio grande", "ester", "clayton", "hackberry",
"middle amana", "new baden", "melville", "rolland colony",
"hannahs mill", "germantown hills", "la fontaine", "aurora",
"green meadows", "kaiminani", "pinecroft", "dawson", "park city",
"hinsdale", "st. meinrad", "kingsland", "powhattan", "bowersville",
"palos verdes estates", "wyandotte", "meigs", "waverly",
"sunol", "arroyo hondo", "outlook", "west pocomoke", "buchtel",
"chatsworth", "smith village", "glenbrook", "rock spring",
"villalba", "bayfield", "waynesfield", "utica", "sunset",
"milford square", "lithium", "swall meadows", "unalaska",
"martinsburg", "ashland", "leawood", "hindsboro", "gray",
"turley", "trimble", "falcon", "linn", "olympia fields",
"mitchell", "mount pleasant mills", "greenville", "park city",
"arkabutla", "new river", "huntsville", "boulder junction",
"potwin", "red lick", "huey", "dougherty", "wadsworth", "grand forks",
"chassell", "edgewood", "lindsay"), lsad = c("25", "57",
"25", "21", "25", "25", "25", "57", "43", "57", "57", "47",
"25", "57", "25", "25", "47", "25", "57", "57", "57", "57",
"21", "25", "57", "57", "43", "25", "43", "57", "57", "25",
"57", "57", "47", "57", "57", "57", "47", "43", "25", "57",
"57", "57", "25", "25", "47", "57", "57", "25", "43", "25",
"43", "25", "57", "57", "57", "57", "57", "47", "25", "43",
"57", "57", "62", "25", "47", "47", "25", "57", "57", "57",
"25", "21", "25", "25", "47", "25", "57", "25", "43", "25",
"47", "25", "57", "25", "57", "57", "57", "25", "57", "25",
"25", "47", "43", "57", "25", "57", "43", "57"), funcstat = c("a",
"s", "a", "a", "a", "a", "a", "s", "a", "s", "s", "a", "a",
"s", "a", "a", "a", "a", "s", "s", "s", "s", "a", "a", "s",
"s", "a", "a", "a", "s", "s", "a", "s", "s", "a", "s", "s",
"s", "a", "a", "a", "s", "s", "s", "a", "a", "a", "s", "s",
"a", "a", "a", "a", "a", "s", "s", "s", "s", "s", "a", "a",
"a", "s", "s", "s", "a", "a", "a", "a", "s", "s", "s", "a",
"a", "a", "a", "a", "a", "s", "a", "a", "a", "a", "a", "s",
"a", "s", "s", "s", "a", "s", "a", "a", "a", "a", "s", "a",
"s", "a", "s"), latitude = c(45.221487, 40.18837, 33.500889,
39.74517, 39.534798, 30.573263, 39.017607, 35.780523, 40.984864,
41.56017, 40.226748, 41.180176, 41.387011, 36.220684, 34.414083,
31.335094, 41.474697, 39.120662, 34.616281, 32.336723, 35.802786,
32.598451, 39.462418, 37.283906, 35.867809, 40.608713, 31.344839,
33.354959, 33.840898, 39.019051, 64.879056, 39.736866, 29.964958,
41.794765, 38.536765, 41.559549, 44.3437, 32.937302, 40.768954,
40.673893, 33.055942, 39.867193, 19.757709, 40.564189, 31.771864,
37.093499, 41.800683, 38.168142, 30.666141, 39.761734, 34.373065,
33.774271, 36.807143, 31.071788, 27.985282, 41.154105, 36.534599,
46.331153, 38.096527, 39.463511, 42.916301, 35.45079, 39.100123,
34.81467, 18.127809, 46.81399, 40.602442, 40.895279, 41.13806,
40.433182, 37.831844, 37.50606, 53.910255, 40.310917, 38.812464,
38.907263, 39.684775, 41.841711, 36.736661, 39.476152, 35.194804,
38.478798, 41.521996, 43.730057, 40.724697, 33.111939, 45.630946,
34.700227, 37.142945, 34.782275, 46.1148, 37.938624, 33.485081,
38.605285, 34.399808, 42.821447, 47.921291, 47.036116, 40.103208,
47.224885), longitude = c(-122.837813, -75.084089, -117.654388,
-77.089213, -74.476099, -97.427116, -94.693955, -81.367835,
-102.262708, -95.994752, -76.902769, -84.736099, -93.272787,
-119.068357, -118.494729, -83.044003, -96.203696, -88.550859,
-82.770697, -90.808692, -94.941358, -114.660588, -75.29244,
-94.926801, -81.044121, -77.23694, -86.46905, -93.497879,
-94.657035, -74.87787, -148.041153, -100.176484, -93.410178,
-91.901539, -89.707193, -71.301933, -96.59226, -84.340945,
-89.462982, -85.722023, -97.509615, -83.945334, -156.001765,
-78.353464, -84.443499, -86.048077, -87.928172, -86.832128,
-98.454026, -95.634011, -83.084305, -118.425754, -94.729305,
-84.092683, -81.625304, -102.762746, -105.666602, -120.092812,
-75.579197, -82.180426, -96.514499, -97.457006, -119.927289,
-85.238869, -66.481897, -90.822546, -83.973881, -97.345349,
-112.028388, -75.405024, -89.88325, -118.642656, -166.529029,
-78.324286, -92.239531, -94.62524, -88.134729, -94.985863,
-107.792147, -94.561898, -78.65389, -91.844989, -87.691648,
-98.029974, -77.026451, -96.110256, -108.925311, -90.121565,
-80.595817, -86.532599, -89.654438, -97.01835, -94.161474,
-89.289973, -97.05148, -77.893875, -97.08933, -88.530745,
-85.737461, -105.152791), designation = c("city", "cdp",
"city", "borough", "city", "city", "city", "cdp", "town",
"cdp", "cdp", "village", "city", "cdp", "city", "city", "village",
"city", "cdp", "cdp", "cdp", "cdp", "borough", "city", "cdp",
"cdp", "town", "city", "town", "cdp", "cdp", "city", "cdp",
"cdp", "village", "cdp", "cdp", "cdp", "village", "town",
"city", "cdp", "cdp", "cdp", "city", "city", "village", "cdp",
"cdp", "city", "town", "city", "town", "city", "cdp", "cdp",
"cdp", "cdp", "cdp", "village", "city", "town", "cdp", "cdp",
"urbana", "city", "village", "village", "city", "cdp", "cdp",
"cdp", "city", "borough", "city", "city", "village", "city",
"cdp", "city", "town", "city", "village", "city", "cdp",
"city", "cdp", "cdp", "cdp", "city", "cdp", "city", "city",
"village", "town", "cdp", "city", "cdp", "town", "cdp")), row.names = c(22769L,
24845L, 3314L, 24015L, 17360L, 28139L, 10085L, 19881L, 3886L,
8750L, 24027L, 20585L, 9362L, 2499L, 3333L, 6041L, 16321L, 6847L,
25249L, 14051L, 22233L, 1210L, 17425L, 10353L, 20053L, 23545L,
253L, 1951L, 22166L, 17386L, 685L, 9771L, 11134L, 9225L, 7386L,
25001L, 25862L, 5663L, 6950L, 8239L, 26555L, 20991L, 6108L, 24388L,
5551L, 10772L, 7056L, 8470L, 27292L, 10202L, 5451L, 3116L, 22660L,
5776L, 5317L, 16546L, 17582L, 29958L, 12103L, 20709L, 8779L,
22515L, 16665L, 5902L, 31901L, 30658L, 21745L, 16574L, 28632L,
24127L, 15046L, 3455L, 930L, 24087L, 14494L, 10016L, 7055L, 8993L,
18048L, 15434L, 19615L, 15043L, 7454L, 25775L, 24194L, 27115L,
15857L, 14038L, 29305L, 276L, 30693L, 10201L, 27863L, 7075L,
22046L, 19267L, 20311L, 12502L, 8093L, 15798L), class = "data.frame")
I would like to calculate the distance between each entry in the city
column using the longitude
and latitude
columns and the gdist
function. I know that the following for
loop works and is easy to read:
dist_list <- list()
for (i in 1:nrow(somewhere)) {
dist_list[[i]] <- gdist(lon.1 = somewhere$longitude[i],
lat.1 = somewhere$latitude[i],
lon.2 = somewhere$longitude,
lat.2 = somewhere$latitude,
units="miles")
}
However: IT TAKES FOREVER to run on the full dataset (31K+ rows)--as in hours. I'm looking for something that will speed up this calculation. I figured something in the split-apply-combine
approach would work well, since I would like to eventually involve a pair of grouping variables, geo_block
and ansi_block
but honestly anything would be better than what I have.
I have tried the following:
somewhere$geo_block <- substr(somewhere$geoid, 1, 1)
somewhere$ansi_block <- substr(somewhere$ansicode, 1, 1)
somewhere <- somewhere %>%
split(.$geo_block, .$ansi_block) %>%
mutate(dist = gdist(longlat = somewhere[, c("longitude", "latitude")]))
But am unsure of how to specify the second set of long-lat inputs outside of the standard for
loop.
My question:
- How do I use the
split-apply-combine
approach to solve this problem withgeo_block
andansi_block
as a grouping variable as above? I would like to return the shortest distance and the name of thecity
and the value ofgeo_block
corresponding to this distance.
All suggestions are welcome. Ideally the desired result would be fairly quick because the actual dataset I'm working with is quite large. Since I'm a bit in the woods here, I've added a bounty to the question to generate a little more interest and hopefully a wide set of potential answers that I can learn from. Thanks so much!
ANSWER
Answered 2021-Nov-12 at 22:48I have such a solution. And I'm surprised myself that I used two loops for
!! Incredibly, I did it. First things first.
My proposal is based on a simplification. However, the mistake you will make at short distances will be relatively small. But the time gain is huge!
Well, I propose to count the distance in Cartesian coordinates, not spherical.
So we're going to need a simple function that computes the Cartesian coordinates based on the two arguments latitude
and longitude
. Here is our LatLong2Cart
feature.
LatLong2Cart = function(latitude, longitude, REarth = 6371) {
tibble(
x = REarth * cos(latitude) * cos(longitude),
y = REarth * cos(latitude) * sin(longitude),
z = REarth *sin(latitude))
}
A short comment right away, my function counts distances in kilometers (after all, it's SI units). If you want you can change it by entering the radius in any other units of miles, yards, feet or whatever else you want.
Let's see how it works. But let me convert yoours data.frame
to tibble
first.
library(tidyverse)
somewhere = somewhere %>% as_tibble()
somewhere %>%
mutate(LatLong2Cart(latitude, longitude))
output
# A tibble: 100 x 12
state geoid ansicode city lsad funcstat latitude longitude designation x y z
1 or 4120100 2410344 donald 25 a 45.2 -123. city -1972. 644. 6024.
2 pa 4280962 2390453 warminster heights 57 s 40.2 -75.1 cdp -4815. -1564. 3867.
3 ca 668028 2411793 san juan capistrano 25 a 33.5 -118. city 485. -3096. 5547.
4 pa 4243944 1214759 littlestown 21 a 39.7 -77.1 borough 350. 2894. 5665.
5 nj 3460600 885360 port republic 25 a 39.5 -74.5 city -1008. -1329. 6149.
6 tx 4871948 2412035 taylor 25 a 30.6 -97.4 city -4237. 160. -4755.
7 ks 2046000 485621 merriam 25 a 39.0 -94.7 city 1435. -686. 6169.
8 nc 3747695 2403359 northlakes 57 s 35.8 -81.4 cdp -2066. -670. -5990.
9 co 839965 2412812 julesburg 43 a 41.0 -102. town 1010. 6223. -915.
10 ia 1909910 2583481 california junction 57 s 41.6 -96.0 cdp 840. 4718. -4198.
# ... with 90 more rows
Once we have the Cartesian coordinates, let's calculate the appropriate distances. I prepared the calcDist
function for this purpose.
calcDist = function(data, key){
if(!all(c("x", "y", "z") %in% names(data))) {
stop("date must contain the variables x, y and z!")
}
key=enquo(key)
n = nrow(data)
dist = array(data = as.double(0), dim=n*n)
x = data$x
y = data$y
z = data$z
keys = data %>% pull(!!key)
for(i in 1:n){
for(j in 1:n){
dist[i+(j-1)*n] = sqrt((x[i]-x[j])^2+(y[i]-y[j])^2+(z[i]-z[j])^2)
}
}
tibble(
expand.grid(factor(keys), factor(keys)),
dist = dist
)
}
This is the place anyway. Right here I used these two for
loops. I hope to be forgiven for that!
Well, let's see if these loops can do something there.
somewhere %>%
mutate(LatLong2Cart(latitude, longitude)) %>%
calcDist(city)
output
# A tibble: 10,000 x 3
Var1 Var2 dist
1 donald donald 0
2 warminster heights donald 4197.
3 san juan capistrano donald 4500.
4 littlestown donald 3253.
5 port republic donald 2200.
6 taylor donald 11025.
7 merriam donald 3660.
8 northlakes donald 12085.
9 julesburg donald 9390.
10 california junction donald 11358.
# ... with 9,990 more rows
As you can see, everything works great. Okay, but how to use it on grouped data? It's nothing hard. Let's group it by states.
somewhere %>%
mutate(LatLong2Cart(latitude, longitude)) %>%
nest_by(state) %>%
mutate(dist = list(calcDist(data, city))) %>%
select(-data) %>%
unnest(dist)
output
# A tibble: 400 x 4
# Groups: state [40]
state Var1 Var2 dist
1 ak ester ester 0
2 ak unalaska ester 9245.
3 ak ester unalaska 9245.
4 ak unalaska unalaska 0
5 al heath heath 0
6 al huntsville heath 12597.
7 al heath huntsville 12597.
8 al huntsville huntsville 0
9 ar stamps stamps 0
10 az orange grove mobile manor orange grove mobile manor 0
# ... with 390 more rows
There is maybe a small redundancy of data here (there are distances from city A to city B and from city B to city A) but the whole thing is relatively easy to implement, which you will probably admit yourself.
Ok. Now it's time to see how it will work when we group as you want by geo_block
and ansi_block
.
somewhere %>%
mutate(LatLong2Cart(latitude, longitude)) %>%
mutate(
geo_block = substr(geoid, 1, 1),
ansi_block = substr(ansicode, 1, 1)) %>%
nest_by(geo_block, ansi_block) %>%
mutate(dist = list(calcDist(data, city))) %>%
select(-data) %>%
unnest(dist)
output
# A tibble: 1,716 x 5
# Groups: geo_block, ansi_block [13]
geo_block ansi_block Var1 Var2 dist
1 1 2 california junction california junction 0
2 1 2 pleasantville california junction 10051.
3 1 2 willacoochee california junction 11545.
4 1 2 effingham california junction 11735.
5 1 2 heath california junction 4097.
6 1 2 middle amana california junction 7618.
7 1 2 new baden california junction 12720.
8 1 2 hannahs mill california junction 11681.
9 1 2 germantown hills california junction 5097.
10 1 2 la fontaine california junction 11397.
# ... with 1,706 more rows
As you can see, there is no problem with that.
Finally, a practical note. When you use this solution, make sure that the grouped variables are not longer than about 20,000 lines. This can lead to memory allocation problems for such large data. Note that then your result will be 20,000 * 20,000 lines which is quite a lot and you may run out of memory at some point. Although the calculations should not take more than a minute.
Please check how it works on your data and give some feedback. I also hope that you liked my solution and that the error of distance resulting from the calculations in Cartesian coordinates will not bother you in any particular way.
QUESTION
Rails 6, Mysql, Ubuntu 20.04 I would like deploy with Capistrano Puma.
I have this error in my website :
We're sorry, but something went wrong.
If you are the application owner check the logs for more information.
Error into my log : puma.error.log :
/home/ubuntu/apps/appex/shared/bundle/ruby/3.0.0/gems/activestorage-
6.1.4.1/lib/active_storage/service/disk_service.rb:14:in `initialize': missing keyword: :root (ArgumentError)
from /home/ubuntu/apps/appex/shared/bundle/ruby/3.0.0/gems/activestorage-6.1.4.1/lib/active_storage/service.rb:61:in `new'
Log Nginx : 2021/11/16 10:05:02 [crit] 3361147#3361147: *6 connect() to unix:///home/ubuntu/apps/appex/shared/tmp/sockets/appex-puma.sock failed (2: No such file or directory) while connecting to upstream, client
ANSWER
Answered 2021-Nov-16 at 13:59That error comes from a misconfigured storage.yml
in your environment. You are missing the root
key.
For more information on how to set up ActiveStorage, read here.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install capistrano
Support
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesExplore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits
Save this library and start creating your kit
Share this Page